Director,  Operational  Test  and  Evaluation 


FY  2014  Annual  Report 


January  2015 


This  report  satisfies  the  provisions  of  Title  10,  United  States  Code,  Section  139. 
The  report  summarizes  the  operational  test  and  evaluation  activities  (including 
live  fir  e  testing  activities)  of  the  Department  of  Defense  during  the  preceding 
fiscal  year. 


Director 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number 


1.  REPORT  DATE 

JAN  2015 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2015  to  00-00-2015 


4.  TITLE  AND  SUBTITLE 

Director,  Operational  Test  and  Evaluation  FY  2014  Annual  Report 


6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Director,  Operational  Test  and  Evaluation, Washington, DC, 20301 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 


13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 


15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 


a  REPORT 

unclassified 


b  ABSTRACT 

unclassified 


c  THIS  PAGE 

unclassified 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

Same  as 

410 

Report  (SAR) 

19a.  NAME  OF 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


F  Y  1  4  INTRODUCTION 


FY  2014 
Annual  Report 


The  purpose  of  operational  testing  is  to  assure  the  Military  Services  field  weapons  that  work  in  combat.  This  purpose  has 
been  codified  in  both  USC  Title  10  and  in  the  Department  of  Defense’s  (DOD)  5000-series  regulations  for  many  years 
without  substantive  alteration.  Operational  testing  is  intended  to  occur  under  “realistic  combat  conditions”  that  include 
operational  scenarios  typical  of  a  system’s  employment  in  combat,  realistic  threat  forces,  and  employment  of  the  systems 
under  test  by  typical  users  (Soldier's)  rather  than  by  hand-picked  or  contractor  crews. 

Thorough  operational  testing  should  be  conducted  prior  to  a  system’s  Full-Rate  Production  decision  or  deployment  to  combat 
in  order  to  inform  acquisition  decision  makers  and  operators  in  an  objective  way  about  how  the  system  will  perform  in  its 
combat  missions.  Under  current  law,  the  Director  of  Operational  Test  and  Evaluation  (DOT&E)  is  required  to  present  his 
opinion  on  whether  the  operational  testing  conducted  prior  to  die  Beyond  Low-Rate  Initial  Production  decision  is  adequate 
or  not.  The  Director  must  consider  all  the  operational  facets  of  a  system’s  employment  in  combat  when  he  determines  what 
constitutes  adequate  operational  testing,  including  the  performance  envelope  the  system  must  be  able  to  achieve,  the  various 
operating  conditions  anticipated  in  a  time  of  war,  and  the  range  of  realistic  operational  threats. 

In  2014, 1  investigated  many  examples  of  recent  programs  across  all  Services  to  identify  common  themes  in  operational 
testing.  These  diemes  illustrate  the  value  that  operational  testing  provides  to  die  Defense  community.  Additionally,  they 
highlight  the  continuing  improvements  we  have  made  in  the  credibility  and  efficiency  of  OT&E  dining  my  tenure.  A  briefing 
covering  diese  six  themes  and  dozens  of  examples  across  all  Sendees  is  posted  on  the  DOT&E  website.1  These  themes 
reveal  a  common  conclusion:  OT&E  provides  value  to  the  Department  by  identifying  key  problems  and  clearly  informing 
warfighters  and  the  acquisition  community  about  the  capabilities  our  combat  systems  do  and  do  not  have.  Furthermore,  we 
are  getting  this  information  now  more  efficiently  and  cost  effectively  than  ever  by  employing  rigorous  scientific  methods  in 
test  planning,  execution,  and  evaluation. 

Identifying  Problems 

One  of  the  primary  purposes  of  operational  testing  is  to  identify  critical  problems  that  can  be  seen  only  when  systems 
are  examined  under  die  stresses  of  realistic  operational  conditions,  prior  to  the  Full-Rate  Production  decision.  This  early 
identification  permits  corrective  action  to  be  taken  before  large  quantities  of  a  system  are  procured  and  avoids  expensive 
retrofit  of  system  modifications.  For  a  recent  example,  operational  testing  of  the  Navy’s  Cooperative  Engagement  Capability 
(CEC)  on  the  E-2D  Hawkeye  aircraft  revealed  several  deficiencies.  The  CEC  created  many  more  dual  tracks  compared  to 
die  baseline  CEC  system,  exhibited  interoperability  problems  widi  the  E-2D  mission  computer,  and  there  was  a  degradation 
in  CEC’s  ability  to  maintain  consistent  air  tracks  compared  to  the  baseline  E-2C  version.  As  a  result  of  these  discoveries 
in  operational  testing,  the  Navy’s  acquisition  executive  decided  to  delay  the  Full-Rate  Production  decision  until  the  root 
causes  for  these  deficiencies  could  be  found  and  fixed.  The  Navy  is  now  implementing  fixes  to  address  these  problems, 
and  operational  testing  will  be  conducted  to  verify  these  fixes  have  corrected  the  problems.  The  value  of  such  testing  is 
abundantly  clear  if  one  considers  the  alternative:  discovering  these  problems  for  die  first  time  in  combat,  when  it  is  too  late 
to  correct  them. 

Fixing,  Not  Testing,  Delays  Programs 

Operational  testing  frequently  reveals  deficiencies  in  a  system  diat  require  time  and  perhaps  also  training  to  correct.  The 
acquisition  executives  who  are  responsible  for  programmatic  decisions  then  have  to  weigh  whether  the  problems  discovered 
are  of  sufficient  magnitude  to  warrant  delays  to  the  program  while  they  are  fixed  (and  re-tested).  The  assertion  that  testing 
causes  programmatic  delays  misses  the  essential  point:  fixing  the  deficiencies  causes  delays,  not  the  testing.  Furthermore, 
taking  die  time  to  correct  serious  problems  is  exactly  what  we  desire  in  a  properly-functioning  acquisition  system;  testing  is 
die  vehicle  by  which  decision  makers  can  be  informed  and  make  decisions  that  will  ultimately  benefit  die  Services  and  the 
Nation. 

This  year,  my  office  updated  a  previous  study  that  we  conducted  with  USD(AT&L)  in  2011  on  the  causes  of  program  delays. 
This  year’s  analysis  examined  case  studies  for  115  acquisition  programs,  which  were  selected  because  they  had  experienced 


1 .  http://www.dote.osd  mil/pub/piesentations/Value_of_OT_Final_Version_8.pdf 
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a  delay  of  6  months  or  more  and  had  a  Full-Rate  Production 
decision  after  2000.  Delays  on  these  programs  ranged  from 
6  months  up  to  15  years,  and  in  some  cases,  programs  were 
cancelled  after  the  delays  (Figure  1  shows  the  distribution  of 
these  delays  for  these  115  programs).  The  reasons  behind  the 
delays  are  varied.  In  most  cases,  the  delay  is  not  due  to  a  single 
reason;  rather  multiple  reasons  led  to  a  delay  (see  Figure  2  and 
Table  1). 

The  study  revealed  that  the  least  common  reason  for  a  delay  was 
a  problem  associated  with  test  conduct.  As  shown  in  Table  1, 
problems  in  test  conduct  occur  in  only  23  percent  of  the  case 
studies,  26  of  115  cases.  Furthermore,  all  programs  that  had 
problems  in  test  conduct  also  had  at  least  one  other  reason  that 
contributed  to  the  delay;  test  conduct,  therefore,  was  never  the 
sole  reason  for  delaying  a  program.  On  the  other  hand,  the  most 
common  reason  that  contributes  to  a  delay  is  a  performance 
problem  discovered  during  developmental  or  operational  testing 
that  must  be  addressed  before  a  program  moves  forward.  A  total 
of  87  of  115  cases  examined  (76  percent)  discovered  system 
performance  problems  during  testing;  38  cases  discovered 
problems  in  developmental  testing  only;  1 7  cases  discovered 
problems  in  operational  testing  only:  and  32  cases  discovered 
problems  in  both  developmental  and  operational  testing. 

Furthermore,  when  examining  the  length  of  the  delay,  no 
statistical  evidence  exists  to  support  the  claim  that  test  conduct 
problems  drive  longer  delays.  Rather,  the  statistics  support 
the  assertion  that  performance  problems  discovered  in  testing 
significantly  affect  the  length  of  the  delay,  not  problems  in 
conducting  the  test.  For  programs  that  discovered  problems 
in  operational  testing,  the  length  of  the  delay  was 
more  significant  than  for  programs  that  discovered 
problems  in  developmental  testing.  This  is  not  a 
surprising  result,  since  problems  seen  for  the  first 
time  in  operational  tests  are  frequently  discovered 
late  in  die  program’s  development,  when  designs 
are  set  and  it  is  more  difficult  and  time  consuming 
to  change  them  to  correct  problems.2  Moreover, 
the  statistical  analysis  revealed  the  largest  drivers 
of  delays  are  whether  the  program  experienced 
manufacturing,  software  development,  or 
integration  problems  and  programmatic  issues. 

A  briefing  witii  more  details  on  diis  analysis  is 
available  on  the  DOT&E  website.3 
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TABLE  1 .  NUMBER  OF  PROGRAMS  AFFECTED  BY  EACH  REASON 

Reason  that  Contributes  to  the  Delay 

Number  of 
Programs  Affected1 

Problems  conducting  the  test:  problems  with  test  resources, 
test  instrumentation,  or  test  execution  that  are  typically  beyond 
the  control  of  the  program  manager 

26 

System  problems  identified  during  testing  that  must  be 
addressed  before  the  program  can  move  forward: 

•  During  developmental  testing  only  (38) 

•  During  operational  testing  only  (1 7) 

•  During  developmental  and  operational  testing  (32) 

87 

Programmatic  issues:  funding,  scheduling,  or  management 
problems 

72 

Manufacturing,  software  development,  integration,  or  quality 
control  problems 

61 

Critical  Nunn-McCurdy  breach2 

34 

1 .  The  total  number  of  programs  affected  is  more  than  1 1 5  because  most  programs  had  more  than 
one  reason  for  a  delay. 

2.  A  critical  Nunn-McCurdy  breach  occurs  when  the  program  acquisition  unit  cost  or  the 
procurement  unit  cost  increases  by  at  least  25  percent  over  the  current  baseline  estimate  or  by 
at  least  50  percent  over  the  original  baseline  estimate. 

2.  DOT&E  employed  a  lognormal  regression  analysis  that 
investigates  the  expected  program  delay  duration  as  a 
function  of  each  of  die  delay  reasons  listed  in  the  table. 

The  analysis  revealed  that  delay  duration  is  statistically 
significantly  affected  by  the  following  factors:  (small 
p- values,  particularly  those  below  0.10,  indicate  that  the 
factor  significandy  affects  the  delay  duration).  P-values 

were  0.08  for  critical  Nunn-McCurdy  breaches;  0.08  for  programmatic  issues;  0.001  for  manufacturing,  software  development,  integration, 
or  quality  control  problems;  and  0. 1 1  for  problems  discovered  in  operational  testing.  Problems  discovered  in  developmental  testing  and 
problems  in  test  conduct  were  not  statistically  significant  factors,  since  their  p-values  were  both  greater  than  0.30. 

3.  2011  Study:  http://www.dote.osdmil/pub/presentations/20110830Program_delays_Nium-McCurdy_final.pdf 
2014  Update:  http://www.dote.osd.mil/pub/presentations/ProgramDelaysBriefing2014_8Aug_Final-77u.pdf 
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So,  to  reiterate,  fixing  problems  discovered  during  testing  causes  program  delays,  not  the  testing  itself. 

In  the  remainder  of  this  introduction,  I  describe  in  more  detail  several  recent  areas  of  focus  for  my  office.  These  include: 

•  My  continued  emphasis  on  the  need  for  statistical  rigor  in  both  the  planning  of  operational  tests  and  the  analysis  of  data 
from  testing. 

•  My  continued  emphasis  on  the  need  to  improve  reliability  of  all  weapon  systems.  I  include  an  assessment  of  new  policies 
on  reliability  growth  and  tracking,  as  well  as  how  the  Department  is  progressing  in  improving  reliability  of  weapon 
systems. 

•  My  new  guidance  on  cybersecurity  testing.  Now  and  in  the  future,  cybersecurity  threats  will  arguably  be  some  of  the 
most  dangerous  threats  our  defense  systems  face.  In  2014, 1  signed  out  guidance  for  testing  the  robustness  of  our  combat 
systems’  abilities  to  withstand  cyber  threats.  In  this  introduction,  I  outline  the  highlights  and  the  importance  of  this 
guidance,  as  well  as  recent  cyber  testing  efforts. 

•  My  emphasis  on  ensuring  adequate  test  resources  are  available  even  when  Department  budgets  are  constrained. 

•  An  assessment  of  problem  discovery  during  testing.  This  section  of  the  report  was  added  in  201 1  based  on  concerns  from 
Congress  that  significant  problems  in  acquisition  programs  are  being  discovered  during  operational  testing  that  arguably 
should  have  been  discovered  in  development  testing  (page  13  in  the  DOT&E  Activity  and  Oversight  section). 


RIGOROUS,  DEFENSIBLE,  EFFICIENT  TESTING 

Since  my  appointment  as  Director,  I  have  required  thorough  operational  tests  that  provide  adequate  information  to 
characterize  system  performance  across  a  variety  of  operational  conditions.  This  information  is  essential  to  my  evaluation 
of  system  operational  effectiveness,  suitability,  and  survivability.  I  have  advocated  the  use  of  scientific  methodologies, 
including  experimental  design  or  design  of  experiments  (DOE)  to  ensure  that  this  characterization  is  done  as  efficiently  as 
possible.  The  methodologies  that  I  have  advocated  for  not  only  provide  a  rigorous  and  defensible  coverage  of  the  operational 
space,  they  also  allow  us  to  quantify  the  trade-space  between  the  amount  of  testing  and  the  precision  needed  to  answer  the 
complex  questions  about  system  performance.  They  allow  us  to  know,  before  conducting  the  test,  which  analyses  we  will 
be  able  to  conduct  with  the  data  and  therefore,  what  questions  about  system  performance  we  will  be  able  to  answer.  Finally, 
they  equip  decision  makers  with  the  analytical  tools  to  decide  how  much  testing  is  enough  in  the  context  of  uncertainty. 

There  has  been  much  progress  in  increasing  the  statistical  rigor  of  test  plans  since  2009.  Over  the  past  several  years,  all  of 
the  Service  Operational  Test  Agencies  (OTAs)  have  implemented  DOE  practices  to  varying  degrees  and  have  offered  training 
to  their  staffs  on  the  statistical  principles  of  DOE.  Additionally,  the  Deputy  Assistant  Secretary  of  Defense  Developmental 
Test  and  Evaluation  (DASD(DT&E))  endorses  these  methods  and  advocates  them  through  his  Scientific  Test  and  Analysis 
Techniques  (STAT)  T&E  Implementation  Plan.  That  office  has  also  overseen  the  conduct  of  the  STAT  in  T&E  Center  of 
Excellence  (COE),  which  employs  qualified  statistics  experts  to  aid  acquisition  program  managers  in  applying  advanced 
statistical  techniques  in  developmental  testing.  However,  these  steps  are  not  enough.  In  the  DOD,  we  acquire  some  of  the 
world’s  most  complex  systems,  but  our  test  and  analysis  capabilities  lag  behind  the  state  of  the  practice,  let  alone  the  state 
of  the  art  for  statistical  techniques.  The  DOD  Test  and  Evaluation  community  should  be  setting  the  standard  for  test  and 
evaluation,  not  struggling  to  apply  methods  that  have  been  applied  for  decades  in  other  test  organizations. 

Moreover,  it  is  not  sufficient  to  only  employ  statistical  methods  in  the  test  design  process;  the  corresponding  analysis 
methods  should  be  employed  in  the  evaluation  of  system  performance,  else  we  risk  missing  important  conclusions.  One 
example  of  the  benefits  of  statistical  analysis  methodologies  was  revealed  during  the  operational  test  of  a  Navy  helicopter 
program,  the  Multi-spectral  Targeting  System,  which  is  intended  to  enable  helicopters  to  target  fast, small-boat  threats  and 
employ  HELLFIRE  missiles  at  safe-standoff  distances.  The  testing  conducted  thoroughly  examined  performance  under 
the  variety  of  operational  and  tactical  conditions  that  a  crew  might  expect  to  encounter,  including  a  variety  of  threat  types 
and  operating  profiles,  as  well  as  engagements  in  different  ocean  and  daylight  conditions.  A  simple  analysis  of  the  results 
combining  all  of  the  data  together  into  a  single  average  (a  particularly  limiting  but  unfortunately  common  analysis  technique) 
suggested  the  system  was  meeting  requirements.  However,  only  when  the  more  complex  and  rigorous  statistical  analysis 
was  employed  did  the  testers  discover  that  the  system  was  significantly  failing  requirements  in  a  subset  of  the  operational 
conditions.  The  unique  set  of  conditions  in  which  performance  was  poor  revealed  a  weakness  in  the  system,  which  can  now 
be  addressed  by  system  developers.  It  is  important  to  note  that  if  DOT&E  had  not  pushed  for  this  rigorous  analysis,  this 
result  would  have  been  missed  completely. 

While  there  has  been  a  lot  of  progress,  much  work  remains.  The  implementation  of  these  techniques  is  still  far  from 
widespread  across  all  DOD  T&E  communities.  Overall,  statistical  analysis  methods  such  as  regression  and  analysis  of 
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variance,  which  supported  the  above  discovery,  are  underused.  Until  they  are  routinely  employed  in  the  analysis  of  T&E 
data,  many  situations  such  as  the  Midti-spectral  Targeting  System  example  are  likely  to  be  missed.  Furthermore,  we  are 
currently  not  leveraging  these  methods  in  a  sequential  fashion  to  improve  knowledge  as  we  move  from  developmental  testing 
to  operational  testing.  Sequential  learning  is  at  the  heart  of  the  experimental  method,  which  all  testing  is  based  on,  and 
we  need  to  employ  such  approaches  in  DOD  T&E.  Doing  so  will  aid  in  improving  our  defense  systems  by  enabling  early 
problem  discovery,  supporting  integrated  testing,  and  improving  our  ability  to  clearly  define  an  adequate  operational  test  that 
avoids  the  unnecessary  expenditure  of  resources. 

DOT&E  Efforts  to  Institutionalize  Test  Science 

Institutionalizing  scientific  approaches  to  testing  in  the  DOD  T&E  community  requires  a  change  from  business  as  usual.  My 
office  has  worked  to  provide  the  motivation,  resources,  education  and  training,  and  overall  support  to  the  T&E  community  to 
make  this  change  possible.  DOT&E  has  worked  to  institutionalize  test  science  principles  by: 

•  Updating  policy  and  guidance  to  reflect  scientific  best  practices 

•  Developing  educational  and  training  resources  to  advance  the  current  workforce 

•  Developing  case  studies  and  examples  to  illustrate  the  value  of  using  such  techniques  in  test  and  evaluation 

•  Forming  advisory  groups  and  a  research  consortium  to  provide  the  T&E  workforce  with  advice,  help  solve  challenging 
problems,  and  develop  the  future  workforce 

A  2013  report  summarized  the  efforts  to  institutionalize  scientific  methods  in  the  DOD  T&E  community  and  discussed 
each  of  these  focus  areas.  This  year,  we  have  continued  to  advance  each  of  those  areas  and  provide  support  for  the  T&E 
community;  our  efforts  are  described  below. 

Policy  and  Guidance 

Both  the  developmental  and  operational  testing  portions  of  the  Interim  DOD  Instruction  5000.02,  “Operation  of  the  Defense 
Acquisition  System,”  now  call  for  a  scientific  approach  to  testing.  The  operational  test  section  calls  for  documenting  each 
of  the  elements  of  the  test  design,  as  well  as  basing  all  test  resources  on  statistically-derived  test  design  and  quantification 
of  risk.  I  have  authored  several  guidance  memos  on  the  use  of  statistical  methods  in  OT&E,  and  the  new  5000.02  guidance 
codifies  the  driving  principles  of  those  guidance  memos  in  DOD  policy. 

In  2014, 1  issued  a  new  guidance  memo  on  the  design  and  use  of  surveys  in  OT&E.  Surveys  provide  valuable  quantitative 
and  qualitative  information  about  the  thoughts  and  feelings  of  operators  and  maintainers  as  they  employ  weapon  systems 
in  an  operationally  realistic  test  environment.  An  objective  measurement  of  these  thoughts  is  an  essential  element  of  my 
evaluation  of  operational  effectiveness  and  suitability.  However,  I  have  noted  that  many  of  the  surveys  used  in  operational 
T&E  are  of  such  poor  quality  they  can  actually  hinder  my  ability  to  objectively  evaluate  the  system.  For  this  reason,  I  issued 
a  policy  memorandum,  based  on  best  practices  from  the  academic  community,  that  provided  concrete  guidance  for  the  use  of 
surveys  in  OT&E.  The  key  elements  included: 

•  Use  surveys  only  when  appropriate;  do  not  ask  operators  about  system  performance  attributes  that  are  more  appropriately 
measured  by  the  testers  (e.g.,  accuracy/timeliness  of  the  system). 

•  Use  the  right  survey  and  leverage  established  surveys  when  appropriate. 

•  Employ  academically-established  best  practices  for  writing  and  administering  surveys. 

1  strive  to  ensure  that  my  guidance  is  widely  available  to  the  T&E  community.  The  DOT&E  website  is  a  convenient  source 
for  all  DOT&E  guidance  memos.4  Additionally,  the  website  has  a  copy  of  the  current  Test  and  Evaluation  Master  Plan 
(TEMP)  Guidebook,  which  is  an  important  resource  for  DOT&E  guidance  on  TEMPs.5  This  guidebook  provides  references 
to  all  guidance  and  policy  memoranda,  describes  in  plain  language  what  I  am  looking  for  in  my  review  of  TEMPs,  and 
provides  several  examples  taken  from  various  TEMPs  that  meet  my  expectations.  The  TEMP  format  was  revised  with  the 
new  5000.02.  New  content  was  added,  including  a  requirements  rationale  and  overview  of  the  concept  of  operations.  My 
staff  is  currently  working  on  the  development  of  the  next  version  of  the  TEMP  Guidebook,  which  I  expect  will  be  available  in 
2015. 

Education  and  Training 

The  use  of  statistical  methods  in  DOD  T&E  has  been  limited  by  inadequate  access  to  education  and  training  opportunities 
by  our  T&E  practitioners.  Many  great  training  opportunities  exist,  and  I  encourage  DOD  leadership  to  make  a  commitment 
to  improving  the  access  of  our  T&E  professionals  to  education  and  training.  Select  members  of  the  workforce  need  to  have 
graduate  degrees  in  fields  related  to  test  science  (statistics,  applied  mathematics,  operations  research,  etc.).  Additionally, 
all  members  of  the  T&E  workforce,  including  the  system  engineers  who  develop  and  test  these  systems  prior  to  formal 

4.  http://www.dote.osd  mil/guidance  html 

5.  http://www.dote.osd  mil/temp-guidebook/index  html 
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developmental  and  operational  testing,  should  have  a  base  level  of  training  in  experimental  design  and  statistical  analysis 
methods.  A  combination  of  both  longer-term  education  and  short-term  training  is  necessary  for  our  test  organizations  to  truly 
improve  the  rigor  of  all  testing.  At  DOT&E,  we  have  developed  custom  training  for  our  action  officers  on  DOE,  reliability, 
survey  design,  and  statistical  analyses.  Additionally,  we  have  developed  advanced  training  materials  for  our  analysts  on 
power  analysis  and  statistical  analysis  methods.  I  am  always  happy  to  share  these  resources  with  the  rest  of  the  T&E 
community  and  welcome  the  participation  of  OTAs  and  other  organizations  in  our  DOT&E  training. 

In  addition  to  providing  training,  DOT&E  is  committed  to  developing  an  online  knowledge  center  for  the  DOD  T&E 
community.  We  have  initiated  development  of  a  web-based  interface  to  this  knowledge  center,  which  will  include  training 
material,  resources  for  best  practices,  tutorials,  and  web-based  tools  related  to  DOE  and  statistical  analysis.  This  website  is 
being  built  in  collaboration  with  many  organizations  across  the  DOD  T&E  community.  An  initial  version  of  the  knowledge 
center  is  scheduled  for  completion  in  2015. 

Case  Studies 

Case  studies  provide  valuable  insight  on  the  application  of  statistical  methods,  including  DOE  in  operational  testing.  Over 
the  past  several  years,  DOT&E  has  developed  many  case  studies  illustrating  the  application  of  DOE  and  statistical  analysis  in 
T&E.  Many  of  these  case  studies  are  summarized  in  previous  DOT&E  publications  (e.g.,  Test  Science  Roadmap  published 
in  2013),  but  new  case  studies  continue  to  be  developed.6  As  these  new  case  studies  have  become  available,  we  have  shared 
them  with  the  OTA  leadership. 

Testing  of  the  complex  systems  the  DOD  T&E  community  encounters  often  requires  non-standard  applications  of  the  tools 
and  methods  currently  available  in  the  statistics  and  test  literature.  DOT&E  has  used  case  studies  to  illustrate  how  advanced 
methods  can  be  used  to  improve  test  outcomes  and  sculpt  existing  methods  to  meet  the  needs  of  the  T&E  community.  One 
example  of  this  occurs  in  the  translation  between  probability-based  metrics  and  more  informative  continuous  metrics,  such  as 
time,  distance,  etc.  Binary  or  probability-based  requirements  such  as  probability-of-detection  or  probability-of-hit,  provide 
operationally  meaningful  and  easy-to-interpret  test  outcomes.  However,  they  are  information-poor  metrics  that  are  extremely 
expensive  to  test.  Having  a  continuous  metric  can  reduce  test  sizes  by  50  percent  or  more  and  provide  more  information  in 
the  analysis,  but  it  can  be  unclear  how  to  convert  between  the  probability  metric  and  the  corresponding  continuous  metric. 
DOT&E  has  developed  several  case  studies  illustrating  this  translation  for  different  test  outcomes.  We  are  now  using 
statistical  cumulative  density  functions  and  censored  data  analyses,  resulting  in  much  more  efficient  tests  that  require  smaller 
sample  sizes  than  would  be  required  to  accurately  measure  the  related  binary  metric. 

Advisory  Groups 

Engagement  with  the  academic  community  and  leading  experts  in  the  field  of  T&E  is  essential  to  advancement  of  these 
rigorous  statistical  techniques  in  DOD  T&E.  In  2014,  DOT&E  renewed  funding  for  the  Test  Science  Research  Consortium 
in  partnership  with  the  Department’s  Test  Resource  Management  Center.  This  multi-year  research  consortium  is  tasked 
with  addressing  the  unique  needs  of  the  T&E  community.  This  consortium  funds  several  graduate-level  research  projects 
on  advanced  statistical  techniques,  enabling  these  projects  to  focus  on  topics  of  benefit  to  the  Department’s  T&E  needs  and 
preparing  a  pipeline  of  students  with  strong  technical  skills  to  learn  about  to  the  Department  and  the  T&E  community.  This 
research  consortium  has  already  produced  several  new  members  of  the  T&E  community  with  advanced  degrees  in  statistics 
and  related  fields. 

Finally,  the  STAT  T&E  COE  has  for  three  years  provided  direct  T&E  support  to  24  program  offices.  The  COE  has  provided 
these  programs  with  direct  access  to  experts  in  test  science  methods,  which  would  otherwise  have  been  unavailable.  I 
have  observed  much  benefit  and  value  from  the  COE’s  engagement  with  programs.  However,  the  COE’s  success  has 
been  hampered,  in  part,  by  unclear  funding  commitments  in  the  out-years.  Furthermore,  the  analysts  are  often  constrained 
to  only  answering  specific,  and  sometimes  narrowly-defined  questions,  as  opposed  to  providing  independent  thought  on 
the  full  content  of  a  program’s  development  test  strategy.  In  the  case  of  the  self-defense  testing  for  the  Air  and  Missile 
Defense  Radar  and  DDG  5 1  Flight  III  Destroyer,  the  COE  analysts  were  constrained  to  constructing  a  test  for  a  limited  set 
of  conditions,  particularly  excluding  the  self-defense  region  near  the  ship  where  the  complex  interactions  between  multiple 
combat  system  components  (missiles,  radars,  and  control  systems)  are  not  known.  Although  the  test  design  provided  was 
robust  for  the  limited  question  asked  of  the  COE,  it  egregiously  missed  the  most  crucial  region  of  the  battlespace,  and  gave 
the  false  impression  that  results  from  such  a  test  design  were  adequate  to  fully  characterize  performance  of  the  combat 
system  and  that  a  self-defense  test  ship  was  unneeded  to  examine  performance  in  the  self-defense  region.  I  will  continue  to 
advocate  that  programs  have  access  to  a  STAT  COE  and  make  use  of  these  excellent  capabilities;  however,  the  COE  must 

6.  http://www.dote.osd  mil/pub/reports/20130711TestScienceRoadmapReport.pdf  and 

http://www.dote.osd  mil/pub/reports/2013071  lAppdxes2theTestScienceRoadmapReport.pdf 
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have  the  ability  to  provide  independent  assessments  to  programs.  Furthermore,  the  COE  needs  to  be  appropriately  funded  to 
be  successful  and  needs  to  expand  in  size  to  aid  program  managers  in  smaller  acquisition  programs  (Acquisition  Category  II 
and  III).  Smaller  programs  with  limited  budgets  do  not  have  access  to  strong  statistical  help  in  their  test  programs  and  cannot 
afford  to  hire  a  full-time  PhD-level  statistician  to  aid  their  developmental  test  program;  having  access  to  these  capabilities 
in  the  STAT  COE  on  an  as-needed  basis  is  one  means  to  enable  these  programs  to  plan  and  execute  more  statistically  robust 
developmental  tests. 


RELIABILITY  ANALYSIS,  PLANNING,  TRACKING,  AND  REPORTING 

I,  and  other  Department  leaders,  have  placed  emphasis  on  improving  the  reliability  of  DOD  systems  via  several  reliability 
improvement  initiatives,  and  I  continue  to  emphasize  the  importance  of  reliability  in  my  assessments  of  operational 
suitability.  There  is  evidence  that  those  systems  that  implement  and  enforce  a  comprehensive  reliability  growth  program  are 
more  likely  to  meet  their  reliability  goals;  however,  test  results  from  the  last  few  years  indicate  the  DOD  has  not  yet  realized 
statistically-significant  improvements  in  the  reliability  of  many  systems. 

The  Department  has  acknowledged  this  poor  track  record  of  meeting  system  reliability  requirements.  In  2011,  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L))  issued  a  Directive  Type  Memorandum 
(DTM  11-003)  on  “Reliability,  Analysis,  Planning,  Tracking,  and  Reporting.”  The  DTM  requires  program  managers  to 
formulate  a  comprehensive  reliability  and  maintainability  program  that  is  part  of  the  systems  engineering  process,  assess  the 
reliability  growth  required  for  the  system  to  achieve  its  reliability  threshold  during  Initial  Operational  Test  and  Evaluation 
(IOT&E),  and  report  the  results  of  that  assessment  to  the  Milestone  Decision  Authority  at  Milestone  C.  To  instantiate 
reliability  reporting  in  support  of  Defense  Acquisition  Executive  Summary  (DAES)  reviews,  DOT&E  has  worked  with 
DOD’s  Systems  Engineering  office  in  USD(AT&L)  to  implement  a  systematic  process  of  tracking  the  reliability  status  of  a 
Major  Defense  Acquisition  Program  (MDAP).  Beginning  with  FY14,  MDAPs  in  system-level  developmental  testing  with  a 
documented  reliability  growth  curve  in  the  Systems  Engineering  Plan  or  TEMP  were  required  to  report  reliability  data  on  a 
quarterly  basis.  At  present,  18  programs  are  actively  reporting  reliability  data  via  this  process,  making  a  system’s  progress 
relative  to  its  expectations  (seen  through  its  reliability  growth  curve)  a  visible  factor  for  the  DAES  process  to  consider. 

While  the  number  of  systems  reporting  these  data  to  DAES  is  increasing,  the  information  is  not  yet  being  used  to  trigger 
programmatic  reviews  or  decision-making. 

Current  Reliability  Trends 

To  better  understand  ongoing  trends  in  reliability,  my  office  has  conducted  a  survey  of  programs  under  DOT&E  oversight  in 
each  of  the  past  six  years  to  determine  the  extent  to  which  reliability-focused  policy  guidance  is  being  implemented  and  to 
assess  whether  it  is  leading  to  improved  reliability.  The  most  recent  survey  focused  on  90  programs  that  either  submitted  a 
TEMP  to  DOT&E  and/or  had  an  operational  test  in  FY13. 

The  survey  results  indicate  that  programs  are  increasingly  incorporating  reliability-focused  policy  guidance.  Since  FY13: 

•  89  percent  of  programs  had  a  reliability  growth  strategy,  with  92  percent  documenting  it  in  the  TEMP. 

•  Likewise,  83  percent  of  programs  incorporated  reliability  growth  curves  into  the  TEMP. 

•  88  percent  of  programs  had  interim  reliability  metrics  prior  to  the  system  requirement. 

•  While  only  28  percent  of  programs  with  FY13  TEMPs  included  a  discussion  of  producer  and  consumer  risk  for  passing  the 
reliability  threshold  in  IOT&E,  this  represents  significant  progress  because  only  one  program  had  done  this  in  the  past. 

Differences  have  been  observed  in  the  implementation  of  policy  across  the  different  services.  The  Army  has  been  a  leader 
at  implementing  reliability  policy  and  the  use  of  reliability  growth  planning  curves,  and  while  the  Air  Force  has  caught  up 
considerably,  many  Navy  programs  have  yet  to  adopt  these  methods.  This  includes  the  use  of  reliability  growth  curves,  the 
use  of  intermediate  goals  based  on  demonstrating  reliability  thresholds  at  operational  test  events,  and  discussing  producer  and 
consumer  risk  (statistical  power  and  confidence)  in  the  TEMP. 

Despite  these  improvements  in  policy  implementation,  we  have  not  observed  a  similarly  improving  trend  in  reliability 
outcomes  at  operational  test  events.  Reliability  growth  curves  are  excellent  planning  tools,  but  programs  will  not  achieve 
their  reliability  goals  if  they  treat  reliability  growth  as  a  “paper  policy.”  Good  reliability  planning  must  be  backed  up  by 
sound  implementation  and  enforcement. 

The  survey  results  indicate  that  two  essential  elements  of  this  implementation  are  1)  including  the  reliability  program  in 
contracting  documents  and  2)  having  reliability-based  test  entrance  criteria.  Programs  that  implemented  at  least  one  of  these 
actions  were  statistically  more  likely  to  meet  their  reliability  requirement  in  operational  testing.  This  is  to  be  expected,  as 
inclusion  of  the  reliability  growth  program  in  contracting  documents  provides  the  government  with  additional  leverage  to 
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ensure  that  contractors  deliver  reliable  systems.  The  survey  results  revealed  that  the  Army  and  Air  Force  have  been  more 
likely  than  the  Navy  to  include  the  reliability  growth  plan  in  contracting  documents  and  meet  entrance  criteria  based  on 
reliability,  availability,  and  maintainability  for  operational  test  events.  Unfortunately,  the  survey  also  revealed  that  it  is  not 
common  practice  for  any  Service  to  implement  these  steps. 

While  following  this  guidance  may  lead  to  improvements  in  reliability  in  the  future,  at  present,  many  programs  still  fail 
to  reach  reliability  goals.  The  reasons  programs  fail  to  reach  these  goals  are  numerous,  but  include  overly-ambitious 
requirements,  unrealistic  assumptions  about  a  program’s  capability  for  reliability  growth,  lack  of  a  design  for  reliability  effort 
prior  to  Milestone  B,  and/or  failure  to  employ  a  comprehensive  reliability  growth  program.  For  example,  the  reliability 
thresholds  for  some  programs  were  unachievably  high  or  disconnected  from  what  was  really  needed  for  the  mission.  In 
some  cases,  a  program’s  reliability  growth  goal,  though  documented  in  a  TEMP  or  SEP,  was  not  supported  by  contractual 
obligations  or  funding.  A  larger  fraction  of  surveyed  programs  met  their  reliability  thresholds  after  fielding  during  Follow-On 
Operational  Test  and  Evaluation  (FOT&E)  (57  percent)  rather  than  before  fielding  during  IOT&E  (44  percent).  I  conclude 
from  this  study  that  although  we  are  in  a  period  of  new  policy  that  emphasizes  good  reliability  growth  principles,  without  a 
consistent  implementation  of  those  principles,  the  reliability  trend  will  remain  flat.  Furthermore,  until  we  as  a  Department 
demonstrate  commitment  to  enforcing  these  principles  by  including  them  in  contracting  documents  and  enforcing  test 
entrance  criteria,  programs  will  have  little  incentive  to  actively  pursue  and  fund  system  changes  that  lead  to  improve 
reliability. 

It  is  also  essential  that  we  collect  enough  information  to  adequately  test  system  reliability.  The  survey  results  showed 
IOT&Es  and  FOT&Es  often  are  not  adequately  sized  to  assess  the  system’s  reliability  requirement  with  statistical  confidence 
and  power.  For  many  programs,  such  testing  is  not  achievable  based  on  concerns  such  as  cost  and  schedule.  In  other  cases, 
the  requirements  were  either  not  testable  or  not  operationally  meaningful.  In  these  cases,  as  always,  my  assessment  of  system 
reliability  was  based  on  how  the  systems’  demonstrated  reliability  would  impact  the  warfighters’  ability  to  complete  their 
mission.  Despite  the  fact  the  survey  revealed  many  operational  tests  are  not  statistically  adequate  to  assess  requirements, 
in  most  of  these  cases,  DOT&E  had  sufficient  data  to  assess  system  reliability  performance.  When  system  reliability 
is  substantially  below  the  requirement,  it  is  possible  to  determine  with  statistical  confidence  the  system  did  not  meet  its 
requirement  with  substantially  less  testing  than  would  otherwise  be  required.  In  other  cases,  other  sources  of  data  can  be 
used.  This  overarching  result  demands  that  we  must  think  about  reliability  testing  differently.  The  next  version  of  my  TEMP 
Guidance  will  include  discussion  on  how  the  TEMP  should  be  used  to  specify  which  data  sources  will  be  used  in  assessing 
system  reliability  at  IOT&E,  as  well  as  the  fidelity  these  sources  must  achieve  to  be  included  in  this  assessment.  This  will 
assist  programs  in  adequately  scoping  IOT&E  and  FOT&E  test  lengths,  helping  them  to  allocate  their  T&E  resources  more 
efficiently. 

National  Academy  of  Sciences  (NAS)  Reliability  Study  Results 

Recently,  the  National  Academy  of  Sciences  (NAS)  published  a  report  on  reliability  and  reliability  growth  for  defense 
systems;  this  report  was  the  result  of  a  study  commissioned  by  myself  and  Mr.  Frank  Kendall,  the  USD(AT&L).7  In 
this  report,  NAS  offered  recommendations  for  improving  the  reliability  of  U.S.  defense  systems.  The  recommendations 
advocated  for  many  of  the  same  principles  that  I  support,  including: 

•  Implementing  reliability  growth  programs  that  include  failure  definitions  and  scoring  criteria  as  well  as  a  structure  for 
reporting  reliability  performance  over  the  course  of  the  acquisition  process. 

•  Using  modern  design-for-reliability  techniques  supported  by  physics  of  failure-based  methods. 

•  Planned  test  lengths  that  are  statistically  defensible. 

NAS  also  suggested  that  these  plans  be  updated  periodically  throughout  the  life  of  the  program,  including  at  major  design 
reviews  and  program  milestones. 

The  NAS  study  addresses  the  need  for  appropriate  reliability  requirements.  NAS  recognizes,  as  I  have  for  years,  the  need 
for  technically-justified,  testable,  mission-relevant  requirements.  These  requirements  must  also  balance  acquisition  costs  and 
lifetime  sustainment  costs.  Systems  that  push  the  limits  of  technical  feasibility  will  be  more  expensive  to  acquire  initially,  but 
may  reduce  lifecycle  costs.  However,  reliability  requirements  that  greatly  exceed  current  capabilities  may  be  unachievable 
and  drive  acquisition  costs  unnecessarily.  As  systems  evolve,  the  requirements  may  need  to  be  updated  as  the  system 
engineering  becomes  more  fully  understood,  but  all  changes  in  these  requirements  should  be  considered  in  the  context  of  the 
mission  impact  of  the  change. 

The  NAS  report  also  points  to  the  importance  of  reliability-focused  contracting.  Making  reliability  a  Key  Performance 
Parameter  on  all  new  systems  and  ensuring  all  proposals  include  explicit  language  designating  funds  and  describing  the 
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design  for  reliability  activities  (including  early  system  reliability  testing)  will  provide  the  DOD  with  leverage  to  ensure 
delivered  systems  are  reliable.  As  mentioned  above,  my  survey  of  acquisition  programs  has  found  that  including  the 
reliability  growth  plan  in  the  contracting  documents  does  indeed  make  systems  more  likely  to  meet  their  reliability  threshold. 
NAS  also  recommends  the  use  of  rigorous  reliability-based  entrance  criteria  prior  to  operational  testing,  stating, 

“Near  the  end  of  developmental  testing,  the  USD(AT&L)  should  mandate  the  use  of  a  full-system, 
operationally-relevant  developmental  test  during  which  the  reliability  performance  of  the  system  will  equal  or  exceed 
the  required  levels.  If  such  performance  is  not  achieved,  justification  should  be  required  to  support  promotion  of  the 
system  to  operational  testing.  ” 

I  have  also  found  that  making  sure  systems  meet  their  entrance  criteria  prior  to  entering  their  IOT&E  makes  them  much  more 
likely  to  perform  well  in  the  operational  test. 

Recent  Lessons  on  Reliability  Requirements 

The  first  step  in  ensuring  reliable  systems  is  to  ensure  that  requirements  are  appropriate.  Sound  reliability  requirements  are 
grounded  in  the  operational  relevance  of  the  missions  the  system  will  support.  They  also  ensure  technical  feasibility  based 
on  existing  systems  and  engineering  limitations,  balance  acquisition  costs  and  sustainment  costs,  and  are  testable.  Two 
recent  examples  have  illustrated  the  tendency  to  develop  reliability  thresholds  for  programs  that  are  unachievably  high  and/or 
disconnected  from  what  is  needed  for  the  mission. 

Three-Dimensional  Expeditionary  Long-Range  Radar  (3DELRR) 

Three-Dimensional  Expeditionary  Long-Range  Radar  (3DELRR)  will  be  the  principal  Air  Force  long-range,  ground-based 
sensor  for  tracking  aircraft  and  other  aerial  targets.  It  replaces  the  aging  TPS-75  radar,  which  is  incapable  of  detecting  some 
current  and  emerging  threats  and  has  become  difficult  and  expensive  to  maintain. 

While  working  with  the  Air  Force  to  develop  plans  to  test  3DELRR,  DOT&E  observed  the  720-hour  Mean  Time  Between 
Critical  Failure  (MTBCF)  appeared  to  be  unnecessarily  high  and  disconnected  from  the  related  availability  requirement, 
set  at  0.947.  When  asked,  the  Air  Force’s  rationale  for  this  requirement  was  initially  presented  as  “the  system  should  be 
operational  for  30  days,  24  hours  per  day,  failure  free.”  The  initial  Service  position  was  that  establishing  an  operational 
availability  (Ao)  of  0.947  with  an  associated  720-hour  MTBCF  would  ensure  the  capability  to  sustain  operations  for  30  days 
in  austere  locations  with  minimum  external  support.  However,  the  probability  of  completing  the  30-day  mission  with  zero 
critical  failures  is  about  0.37,  assuming  a  system  MTBCF  of  720  hours.  Achieving  mission  reliability  values  higher  than  0.37 
would  require  very -high  MTBCF  values;  requiring  a  90-percent  probability  of  completing  a  30-day  mission  without  failure 
would  require  an  MTBCF  of  over  6,800  hours.  A  lower  MTBCF  of  300  hours  would  provide  availability  near  0.90,  with 
each  100  hours  of  reliability  adding  just  a  fraction  to  the  overall  Ao.  Based  on  these  observations,  DOT&E  recommended 
the  Service  review  the  reliability  requirement  to  determine  if  an  MTBCF  of  720  hours  was  truly  needed  to  achieve  reasonable 
reliability.  Additionally,  DOT&E  recommended  that  once  the  reliability  requirement  was  validated,  the  Service  should 
implement  a  design  for  a  reliability  program  and  implement  a  reliability  growth  program. 

After  multiple  discussions  and  review  of  the  logistics  supply  concept,  as  well  as  the  concept  of  operations  for  completing 
missions,  the  Air  Force  recognized  that  a  720-hour  MTBCF  was  not,  in  fact,  required.  After  further  review,  the  Service  set 
the  requirement  that  the  system  will  achieve  an  MTBCF  of  495  hours  by  the  end  of  government-conducted  developmental 
T&E,  along  with  an  Ao  of  0.947.  Furthermore,  the  Air  Force  designed  an  acceptable  reliability  growth  program  that  adheres 
to  best  practices,  and  DOT&E  approved  the  program’s  TEMP  for  Milestone  B. 

Ground/Air  Task  Oriented  Radar  (G/ATOR) 

The  Marine  Corps’  Ground/Air  Task  Oriented  Radar  (G/ATOR)  is  a  phased  array,  multi-role  radar  that  is  being  designed  to 
be  used  initially  in  the  air  surveillance  and  air  defense  roles,  with  follow-on  capabilities  that  will  be  developed  to  support 
the  ground  weapon  locating/counter-targeting  and  air  traffic  control  mission  areas.  During  a  recent  operational  assessment 
period,  G/ATOR  met  its  Ao  Key  Performance  Parameter;  however,  Key  System  Attributes  reflecting  system  reliability  were 
well  below  thresholds.  DOT&E  issued  an  operational  assessment  report,  as  well  as  a  separate  memorandum  discussing  the 
program’s  proposed  reliability  growth  plans  in  the  related  draft  TEMP,  that  again  noted  several  problems  with  the  system’s 
reliability  and  growth-planning  assumptions.  As  a  result  of  G/ATOR  not  meeting  planned  reliability  growth  milestones, 
and  with  no  clear  means  to  grow  the  reliability  to  that  required  and  maintain  program  timelines,  the  Navy  stood  up  a  “Blue 
Ribbon  Panel”  made  of  Department  experts  and  stakeholders  (including  DOT&E)  to  assess  the  program’s  ability  to  achieve 
current  reliability  threshold  requirements.  The  panel’s  findings  included: 

•  The  Mean  Time  Between  Operational  Mission  Failure  reliability  threshold  requirement  is  disconnected  from  the  mission 
(not  operationally  relevant). 
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•  There  is  no  clearly  defined  G/ATOR  system  definition  for  government-  and  contractor-furnished  equipment. 

•  The  rationale  for  excluding  certain  government-furnished  equipment  from  reliability  calculations  is  ambiguous. 

•  There  is  no  closed  loop  in  the  failure  reporting,  analysis,  and  corrective  action  system;  specifically,  the  program’s  Failure 
Review  Board  determines  which  failures  are  valid  but  does  not  formally  adjudicate  the  effectiveness  of  corrective  actions. 

•  Reliability  growth  planning  models  used  optimistic  planning  factors  and  growth  curves,  and  were  based  on  Mean  Time 
Between  Operational  Mission  Failure/Mean  Time  Between  Failure  initial  values  that  were  not  previously  realized  during 
testing. 

•  Definitions  for  failures  and  operating  time  during  previous  developmental  test  are  not  consistent. 

The  findings  were  recently  briefed  to  the  Milestone  Decision  Authority  for  G/ATOR.  Recommendations  to  the  above  findings 
are  currently  under  review. 

Both  the  3DELLR  and  G/ATOR  programs  reveal  the  need  to  carefully  consider  the  reliability  requirements  in  relation  to  what 
is  essential  for  completing  a  mission  within  the  Services’  own  concepts  of  operations.  Acting  on  the  recommendations  of  the 
NAS  study,  as  well  as  those  others  and  1  have  stated,  will  ensure  programs  not  only  are  successful  in  achieving  their  reliability 
goals,  but  also  that  the  goals  are  realistic  and  achievable. 


CYBERSECURITY  OPERATIONAL  TESTING  AND  ASSESSMENTS  DURING  EXERCISES 

Cyber  adversaries  have  become  as  serious  a  threat  to  U.S.  military  forces  as  the  air,  land,  sea,  and  undersea  threats  represented 
in  operational  testing  for  decades.  Any  electronic  data  exchange,  however  brief,  provides  an  opportunity  for  a  determined 
and  skilled  cyber  adversary  to  monitor,  interrupt,  or  damage  information  and  combat  systems.  The  DOD  acquisition  process 
should  deliver  systems  that  provide  secure  and  resilient  cyber  capabilities;  therefore,  operational  testing  must  examine 
system  performance  in  the  presence  of  a  realistic  cyber  threat.  My  assessment  of  operational  effectiveness,  suitability,  and 
survivability  is  determined  in  part  by  the  results  of  this  crucial  testing. 

During  2014,  cybersecurity  testing  of  more  than  40  systems  showed  improvements  must  occur  to  assure  secure  and  resilient 
cyber  capabilities.  One  important  conclusion  from  my  2014  review  of  DOD  programs  was  that  operational  testing  still  finds 
exploitable  cyber  vulnerabilities  that  earlier  technical  testing  could  have  mitigated.  These  vulnerabilities  commonly  include 
unnecessary  network  services  or  system  functions,  as  well  as  misconfigured,  unpatched,  or  outdated  software,  and  weak 
passwords.  Developmental  testing  over  the  course  of  the  program,  including  the  process  to  grant  a  system  the  authority  to 
operate  on  DOD  networks,  could  have  found  most  of  these  vulnerabilities;  yet,  such  vulnerabilities  are  still  found  as  late  as 
during  the  IOT&E.  My  review  of  these  systems  also  identified  the  need  to  increase  the  participation  of  network  defenders 
and  assessment  of  mission  effects  during  threat-representative,  adversarial  assessments. 

In  August  2014, 1  published  updated  policy  and  procedures  for  cybersecurity  assessments  in  operational  T&E;  the  new 
guidance  specifies  that  operational  testing  should  include  a  cooperative  vulnerability  assessment  phase  to  identify  system 
vulnerabilities  followed  by  an  adversarial  assessment  phase  to  exploit  vulnerabilities  and  assess  mission  effects.8  The 
adversarial  assessment  phase  includes  system  users  and  network  defenders  to  detect  the  adversarial  actions,  react  to  those 
actions,  and  restore  the  system  to  full/degraded  operations  after  a  cyber-attack.  My  office  continues  to  emphasize  the 
need  to  assess  the  effects  of  a  debilitating  cyber-attack  on  the  users  of  these  systems  so  that  we  understand  the  impact  to 
a  unit’s  mission  success.  A  demonstration  of  these  mission  effects  are  often  not  practicable  during  operational  testing  due 
to  operational  safety  or  security  reasons.  1  have  therefore  advocated  that  tests  use  simulations,  closed  environments,  cyber 
ranges,  or  other  validated  and  operationally  representative  tools  to  demonstrate  the  mission  effects  resulting  from  realistic 
cyber-attacks. 

Representative  cyber  environments  hosted  at  cyber  ranges  and  labs  provide  one  means  to  accomplish  the  above  goals.  Such 
cyber  ranges  and  labs  provide  realistic  network  environments  representative  of  warfighter  systems,  network  defenses,  and 
operators,  and  they  can  emulate  adversary  targets  and  offensive/defensive  capabilities  without  concern  for  harmful  effects  to 
actual  in-service  systems/networks.  For  several  years,  my  office  has  proposed  enhancements  to  existing  facilities  to  create 
the  DOD  Enterprise  Cyber  Range  Environment  (DECRE),  which  is  comprised  of  the  National  Cyber  Range  (NCR);  the  DOD 
Cybersecurity  Range;  the  Joint  Information  Operations  Range;  and  the  Joint  Staff  J-6  Command,  Control,  Communications, 
and  Computers  Assessments  Division.  The  need  and  use  of  these  resources  is  beginning  to  outpace  the  existing  DECRE 
capabilities.  As  an  example,  the  NCR  experienced  a  substantial  increase  in  customers  in  FY14,  and  the  Test  Resource 
Management  Center,  which  oversees  the  NCR,  has  initiated  studies  to  examine  new  capabilities  to  further  expedite  the 
planning,  execution,  and  sanitization  of  NCR  events. 

8.  http://www.dote.osd  mil/pub/policies/2014/8-l-14_Procs_for_OTE_of_Cybersec_in_Acq_Progs(7994).pdf 
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Also  in  2014,  my  office  conducted  16  cybersecurity  assessments  in  conjunction  with  Combatant  Command  and  Service 
exercises.  A  notable  improvement  over  previous  years  was  the  increased  participation  of  higher-echelon  computer  network 
defense  service  providers  and  local  defenders,  resulting  in  a  more  comprehensive  assessment  of  cyber  defensive  postures. 
Despite  the  improved  defenses,  my  office  found  that  at  least  one  assessed  mission  during  each  exercise  was  at  high  risk  to 
cyber-attack  from  beginner  to  intermediate  cyber  adversaries.  1  have  placed  emphasis  on  helping  Combatant  Commands 
and  Services  mitigate  and  reduce  those  persistent  cybersecurity  vulnerabilities  observed  from  assessment  to  assessment.  My 
continuing  focus  is  on  finding  problems,  providing  information  and  assistance  to  understand  and  fix  problems,  and  following 
up  to  verify  cybersecurity  status  and  ability  to  conduct  operations  in  a  contested  cyberspace  environment.  At  the  request  of 
several  Combatant  Commands,  1  have  implemented  more  frequent  operational  site  assessments  during  day-to-day  operations 
on  live  networks  to  provide  feedback  on  specific  areas  of  interest  such  as  status  of  patching  or  defense  against  specific  attacks 
(e.g.,  phishing)  and  cybersecurity  implications  of  physical  security.  Additional  continuing  efforts  include  working  with  the 
intelligence  community  to  improve  cyber  threat  realism,  and  to  develop  a  persistent  cyber  opposition  force  with  the  capability 
to  operate  across  several  Combatant  Commands. 


TEST  RESOURCES 

Adequate  funding  of  test  resources  remains  a  crucial  aspect  to  fielding  weapons  that  work.  My  office  continues  to  monitor 
DOD  and  Service-level  strategic  plans,  investment  programs,  and  resource  management  decisions  to  ensure  the  Department 
maintains  the  capabilities  necessary  for  adequate  and  realistic  operational  tests.  I  have  continued  to  emphasize  the  need 
for  these  resources  despite  the  constrained  fiscal  environment.  There  are  some  who  argue  that  in  a  constrained  fiscal 
environment,  particularly  in  the  face  of  sequestration,  all  testing  should  be  cut  commensurate  with  cuts  in  program  budgets. 
That  is,  if  the  Department’s  budgets  are  reduced  by  20  percent,  then  testing  should  also  be  reduced  by  20  percent.  Yet,  we 
are  fielding  the  weapons  that  are  developed  to  satisfy  100  percent  of  their  concepts  of  operation  against  100  percent  of  the 
actual  threat.  In  particular,  what  constitutes  adequate  operational  testing  under  realistic  combat  conditions  is  determined  not 
by  fiscal  constraints,  but  by  our  war  plans  and  the  threats  we  face — the  enemy  (always)  gets  a  vote.  It  would  therefore,  be 
a  false  economy  and  a  disservice  to  the  men  and  women  we  send  into  combat  to  make  arbitrary  budget-driven  reductions  to 
either  developmental  or  operational  testing. 

The  T&E  Resources  section  of  this  Annual  Report  details  the  projects  and  issues  on  which  1  am  most  concerned  or  focused. 
Of  particular  note  this  year  is  that  I  remain  concerned  about  the  substantial  year-after-year  staffing  reductions  taken  by  the 
Army  T&E  Executive  and  his  office  within  the  Office  of  the  Deputy  Under  Secretary  of  the  Army,  as  well  as  reduction 
in  staff  levels  in  both  the  Army  Operational  Test  Command  and  the  Army  Evaluation  Center.  These  reduced  staff  levels 
will  cause  delays  to  developmental  and  operational  testing,  the  inability  to  conduct  simultaneous  operational  test  events, 
and  longer  timelines  for  the  release  of  test  reports.  Furthermore,  the  Commander,  Operational  Test  and  Evaluation  Force 
continues  to  try  to  enhance  his  workforce  by  growing  in-house  technical  talent  and  hiring  in  personnel  with  advanced 
degrees. 

As  the  Department  moves  forward  in  considering  important  test  resource  infrastructure  and  investments  in  the  face  of 
constrained  budgets,  1  will  continue  to  advocate  for  the  need  for  the  most  crucial  test  assets.  These  include: 

•  The  need  for  an  Aegis-capable  self-defense  test  ship  to  test  Ship  Self-Defense  Systems’  performance  in  the  final  seconds  of 
the  close-in  battle  and  to  acquire  sufficient  data  to  accredit  ship  self-defense  modeling  and  simulation  test  beds.  (While  the 
Navy  recognizes  the  capability  as  integral  to  the  test  programs  for  certain  weapons  systems  (the  Ship  Self-Defense  System, 
Rolling  Airframe  Missile  Block  2,  and  Evolved  SeaSparrow  Missile  Block  1)  and  ship  classes  (LPD-17,  LF1A-6,  Littoral 
Combat  Ship,  LSD  41/49,  DDG  1000,  and  CVN-78),  the  Navy  has  not  made  a  similar  investment  in  a  self-defense  test 
ship  equipped  with  an  Aegis  Combat  System,  Air  and  Missile  Defense  Radar,  and  Evolved  SeaSparrow  Missile  Block  2 
for  adequate  operational  testing  of  the  DDG  51  Flight  III  Destroyer  self-defense  capabilities.) 

•  The  DECRE,  as  discussed  above. 

•  The  electronic  warfare  infrastructure,  which  the  Department  is  funding  and  for  which  some  slow  progress  is  being 
realized. 

•  The  electronic  warfare  assets  for  anti-ship  cruise  missile  seeker  emulation  and  the  jamming  simulators  for  the  assessment 
of  Army  communications  networks. 

Other  resource  needs  that  I  consider  crucial  for  the  Department  to  pursue  are  detailed  in  the  T&E  Resources  section  of  this 
report  (page  339). 
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SUMMARY 

Since  my  first  report  to  you  in  2009,  we  have  made  progress  increasing  the  scientific  and  statistical  rigor  of  operational 
T&E;  there  is  much  work  to  be  done,  however,  since  the  Department’s  test  design  and  analysis  capabilities  lag  behind  the 
state  of  the  practice.  Additionally,  we  have  focused  attention  on  reliability  design  and  growth  testing,  and  in  improving 
cybersecurity  operational  testing.  Operational  testing  continues  to  be  essential  to  characterize  system  effectiveness  in  combat 
so  well-informed  acquisition  and  development  decisions  can  be  made,  and  men  and  women  in  combat  understand  what  their 
equipment  and  weapons  systems  can  and  cannot  do.  1  submit  this  report,  as  required  by  law,  summarizing  the  operational  and 
live  fire  T&E  activities  of  the  DOD  during  Fiscal  Year  2014. 


Director 
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F  Y  1  4  DOT&E  ACTIVITY  AND  OVERSIGHT 


FY1 4  Activity  Summary 


DOT&E  activity  for  FY14  involved  oversight  of  309  programs, 
including  26  Major  Automated  Information  Systems.  Oversight 
activity  begins  with  the  early  acquisition  milestones,  continues 
through  approval  for  full-rate  production,  and,  in  some  instances, 
during  full  production  until  removed  from  the  DOT&E  oversight 
list. 

Our  review  of  test  planning  activities  for  FY14  included  approval 
of  43  Test  and  Evaluation  Master  Plans  (TEMPs);  3  Test  and 
Evaluation  Strategies;  90  Operational  Test  Plans;  5  Live  Fire  Test 
and  Evaluation  (LFT&E)  Strategies  and  Management  Plans;  and 
disapproval  of  the  following  5  TEMPs  and  5  Test  Plans: 

•  Remote  Minehunting  System  (RMS)  Test  and  Evaluation 
Master  Plan  (TEMP),  TE1N  1520,  Revision  D,  Change  1 

•  Littoral  Combat  Ship  (LCS)  Test  and  Evaluation  Master  Plan 
(TEMP) 

•  Surface  Electronic  Warfare  Improvement  Program  (SEWIP) 
Block  3  Test  and  Evaluation  Master  Plan  (TEMP) 

•  Distributed  Common  Ground  System  -Army  (DCGS-A)  Test 
and  Evaluation  Master  Plan  (TEMP) 

•  Mobile  Landing  Platform  (MLP)  Test  and  Evaluation  Master 
Plan  (TEMP) 

•  Global  Command  and  Control  System  -  Joint  (GCCS-J) 
Version  4.3  Follow-on  Operational  Test  and  Evaluation 
(FOT&E)  Plan 

•  TRICARE  Online  (TOL)  Blue  Button  and  Data  Federation 
Operational  Assessment  (OA)  Test  Plan 

•  Remote  Minehunting  System  OA  Test  Plan 

•  SEWIP  Initial  Operational  Test  and  Evaluation  (IOT&E)  Test 
Plan 

•  Automated  Identification  Biometric  System  (ABIS)  1 .2 
Limited  User  Test  (LUT)  Plan 


In  FY14,  DOT&E  prepared  for  the  Secretary  of  Defense  and 
Congress:  6  IOT&E  reports,  5  Early  Fielding  Reports,  5  FOT&E 
reports,  1  LFT&E  report,  1  OT&E  report,  1  OA  report,  2  special 
reports,  and  the  Ballistic  Missile  Defense  program’s  FY13 
Annual  Report.  DOT&E  also  prepared  and  submitted  numerous 
reports  to  the  Defense  Acquisition  Board  (DAB)  principals  for 
consideration  in  DAB  deliberations.  Additional  FY14  DOT&E 
reports  that  did  not  go  to  Congress  included:  10  Cybersecurity 
reports,  1  Early  Fielding  report,  1  Early  OA  report,  3  FOT&E 
reports,  3  Force  Development  Evaluation  reports,  1  IOT&E 
report,  1  LFT&E  report,  11  Operational  Assessment  reports, 

2  OT&E  reports,  and  5  special  reports. 

During  FY14,  DOT&E  met  with  Service  operational  test 
agencies,  program  officials,  private  sector  organizations,  and 
academia;  monitored  test  activities;  and  provided  information  to 
the  DAB  committees  as  well  as  DAB  principals,  the  Secretary 
and  Deputy  Secretary  of  Defense,  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics,  the  Service 
Secretaries,  and  Congress.  Active  on-site  participation  in,  and 
observation  of,  tests  and  test-related  activities  are  a  primary 
source  of  information  for  DOT&E  evaluations.  In  addition  to 
on-site  participation  and  local  travel  within  the  National  Capital 
Region,  approximately  930  trips  supported  the  DOT&E  mission. 

Security  considerations  preclude  identifying  classified  programs 
in  this  report.  The  objective,  however,  is  to  ensure  operational 
effectiveness  and  suitability  do  not  suffer  due  to  extraordinary 
security  constraints  imposed  on  those  programs. 


TEST  AND  EVALUATION  MASTER  PLANS  /  STRATEGIES  APPROVED 


Aegis  Cruiser  and  Destroyer  Program  TEMP  CNO  Project  Number  1669 
Revision  1 

AH-64ETEMP  Net  Ready  Annex. 

Air  Operations  Center -Weapon  System  (AOC-WS)  Increment  10  2  TEMP 

AN/AAQ-24A  (V)  25  Department  of  the  Navy  Large  Aircraft  Infrared 
Countermeasures  (DoN  LAIRCM)  in  a  Pod  Configuration  (Guardian  Pod) 
and  AN/AAQ-24B  (V)  25  DoN  LAIRCM  Advanced  Threat  Warning  (ATW) 
TEMP 

AN/SQQ-89A(V)1 5  Surface  Ship  Undersea  Warfare  (USW)  Combat  System 
Advanced  Capability  Build  (ACB)TEMP 

Armored  Multi-Purpose  Vehicle  (AMPV)TEMP 
Assault  Amphibious  Vehicle  Survivability  UpgradeTEMP 

C-5  Core  Mission  Computer  and  Weather  Radar  (CMC/Wx  Radar)  Program 
TEMP 


Cartridge  7.62  Millimeter:  Ball,  M80A1  TEMP  Update 
Combat  Rescue  Helicopter  (CRH)  TEMP  for  Milestone  B 
Common  Aviation  Command  and  Control  System  (CAC2S)TEMP 

Consolidated  Afloat  Networks  and  Enterprise  Services  (CANES)  TEMP 
Approval 

Cougar  Family  of  Vehicles  Enduring  Fleet  Modifications  TEMP  Addendum 
Defense  Agencies  Initiative  (DAI)  Increment  2  TEMP  for  Milestone  B 
Defense  Enterprise  Accounting  and  Management  System  (DEAMS)TEMP 

Department  of  Defense  Healthcare  Management  System  Modernization 
(DHMSM)  Program, Test  Strategy 

Distributed  Common  Ground/Surface  System  (DCGS-MC)  -  Marine  Corps 
Geospatial  Intelligence  TEMP 

E-2D  Advanced  Hawkeye  (AHE)  Revision  DTEMP 
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Enhanced  Polar  System  (EPS)  TEMP 

E-2DTEMP  number  1654  Revision  D 

Family  of  Light  Armored  Vehicles  (FOLAV)  TEMP 

Global  Combat  Support  System  -  Joint  (GCSS-J)  Milestone  B/CTEMP 

Global  Combat  Support  System  -  Marine  Corps/Logistics  Chain 
Management  (GCSS-MC/LCM)  Increment  1  TEMP 

Indirect  Fire  Protective  Capability  Increment  2-1  (IFPC  Inc  2-1)  Test  and 
Evaluation  Strategy) 

Integrated  Personnel  and  Pay  System  -  Army  (IPPS-A)  Increment  I 
Milestone  CTEMP 

Joint  Battle  Command  -  Platform  (JBC-P)TEMP 

Joint  Battle  Command  -  Platform  (JBC-P)TEMP  changes,  Version  3.4 

Key  Management  Infrastructure  (KMI)  Increment  2,  Spiral  2 TEMP 
Addendum 

M829E4  (1 20  mm:  Armored  Piercing,  Fin  Stabilized,  Discarding 
Sabot  -Tracer,  Kinetic  Energy  Cartridge)  System  Live  Fire  Test  and 
Evaluation  Strategy 

M829E4  Abrams  Tank  Main  Gun  Round  TEMP 
M982E1  Excalibur  Increment  IbTEMP 

Multifunctional  Information  Distribution  System  (MIDS),  Joint  Tactical 
Radio  System  (JTRS),  Concurrent  Multi-Netting  Four  (CMN-4)  Full 
Deployment  TEMP  Annex  J 


Nett  Warrior  TEMP 

Next  Generation  Chemical  Detector  (NGCD  1 )  Detector  Alarm,  ACAT II 
TEMP,  Version  1.1  Supporting  Milestone  A 

Next  Generation  Chemical  Detector  (NGCD  3)  Sample  Analysis,  ACAT  II 
TEMP,  Version  1.1  Supporting  Milestone  A 

Next  Generation  Chemical  Detector  (NGCD)  Increment  2  Survey  Detector 
TEMP 

P-8A  Poseidon  Multi-mission  Maritime  Aircraft  (MMA)  Increment  2  and 
Full-Rate  Production  (INC  2/FRP)  (ACAT  1DTEIN  1652)  TEMP 

P-8A  Poseidon  Multi-mission  Maritime  Aircraft  (MMA)  Increment  3 
(Pre-MDAP-TEIN  181 3)  TEMP 

Patriot  TEMP 

QF-16  Full  Scale  Aerial  Target,  Version  2.2kTEMP 

RQ-7BV2 Tactical  Unmanned  Aircraft  System  (TUAS)  Acquisition  Category 
(ACAT)  II  (Shadow)  TEMP 

Small  Tactical  UAS  (STUAS)  (RQ-21 A  Blackjack)  TEMP  change 
Three-Dimensional  Expeditionary  Long-Range  Radar  (3DELRR) TEMP 
UFI-60L  Digital  Black  Flawk  FlelicopterTEMP 

Warfighter  Information  Network -Tactical  (WIN-T)  Increment  2  (Inc  2) 
TEMP 

Warfighter  Information  Network -Tactical  (WIN-T)  Increment  2  (Inc  2) 
TEMP  Addendum 


OPERATIONAL  TEST 


AC-130J  Precision  Strike  Package  Operational  Assessment  (OA)Test  Plan 

Advanced  Extremely  High  Frequency  (AEHF)  Satellite  Communications 
(SATCOM)  System  Multi-Service  Operational  Test  and  Evaluation  (MOT&E) 
Operational  Test  Agency  (OTA)  Test  Plan 

Aegis  Baseline  9A  Maintenance  Demonstration  (M-DEMO)  OT-C2A1  Data 
Management  and  Analysis  Plan  (DMAP)  Defense  Enterprise  Accounting 
and  Management  System  (DEAMS)  Financial  Cyber  Readiness  Assessment 
Plan 

Aegis  Weapon  System  (AWS)  Baseline  9A  (BL  9A)  Integrated  Testing  (ITT) 
on  Air  Defense  Cruiser  Data  Management  and  Analysis  Plan  (DMAP) 

Aegis  Weapon  System  (AWS)  Baseline  9A  (BL9A)  Air  Defense  Cruiser 
Integrated  Test  DMAP  Change  3 

Air  Intercept  Missile-9X  (AIM-9X)  Block  ii  (U)Test  Plan  Modifications 

Amphibious  Assault  Ship  Replacement  LHA-R  Flight  1  Early  OATest  Plan 
for  CNO  Project  No.  1 697 

AN/AAQ-24A  (V)  25  Department  of  the  Navy  Large  Aircraft  Infrared 
Countermeasures  (DoN  LAIRCM)  in  a  Pod  Configuration  (Guardian  Pod) 
and  AN/AAQ-24B  (V)  25  DoN  LAIRCM  Advanced  Threat  Warning  (ATW) 
Systems  Integrated  Evaluation  Framework  (IEF) 

AN/PRC-1 1 7G  Radio  Operational  Test  Plan 

AN/SQQ-89A(V)  15  Surface  Ship  Undersea  Warfare  (USW)  Combat  System 
Program  lOT&ETest  Plan  for  CNO  Project  No.  0802-02 

AN/TPQ-53  Target  Acquisition  Radar  System  Initial  Operational  Test  Plan 
Apache  AH-64E  FOT&E  I  OTA  Test  Plan 

Automated  Identification  Biometric  System  (ABIS)  1.2  Limited  User  Test 
(LUT)  Plan 

B-2  IOT&E  Plan  Deviation 


PLANS  APPROVED 

Ballistic  Missile  Defense  System  (BMDS)  Integrated  Master  Test  Plan  (IMTP) 
v14.1  andv14.1  Annex 

C-5  Reliability  Enhancement  &  Re-Engining  Program  (RERP)  Operational 
Flight  Program  (OFP)  3.5  Force  Development  Evaluation  (FDE)Test  Plan 

CENTCOM  AOR  14-1  (SOCCENT)  Final  Assessment  Plan 
Command  Post  of  the  Future  (CPoF)  LUT  OTA  Test  Plan 
Common  Aviation  Command  and  Control  System  (CAC2S)  OATest  Plan 

Consolidated  Afloat  Networks  and  Enterprise  Services  (CANES)  lOT&ETest 
Plan 

Consolidated  Afloat  Networks  and  Enterprise  Services  (CANES)  lOT&ETest 
Plan  Cyber  Security  Annex 

Cooperative  Engagement  Capability  (CEC)  DMAP  for  the  Maintainability 
Demonstration  (M-Demo)  PhaseofOT-DIA 

Countermeasure  Anti-Torpedo  (CAT)  and  Torpedo  Warning  System  (TWS) 
Quick  Reaction  Assessment  (QRA)  DMAP 

Defense  Enterprise  Accounting  and  Management  System  (DEAMS)  IOT&E 
Test  Plan 

Distributed  Common  Ground  System  -  Marine  Corps  (DCGS-MC)  IOT&E 
Plan 

DOD  Teleport  Generation-3,  Phase  1  (G3P1)  OT&E  Plan 
DOD  Teleport  Generation-3,  Phase-2  (G3P2)  OT&E  Plan 
E-2D  FOT&E  Test  Plan 

Enhanced  Combat  Helmet  (ECH)  Lot  Acceptance  Testing  (LAT)  Marine 
Corps  Operational  Test  and  Evaluation  Activity  Observation  Plan 

EPIC  GUARDIAN  14  Information  Assurance  and  Interoperability 
Assessment  Plan 
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Excalibur  Increment  lb  IOT&E  OTA  Test  Plan 

F/A-18E/F  Infrared  Search  and  Track  System  (IRST)  Block  I  OATest  Plan 

F/A-18E/F  Infrared  Search  and  Track  System  (IRST)  Block  I  OA  Phase  II  Test 
Plan 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T)  (Raytheon 
Variant)  OA-1  Plan 

Global  Combat  Support  System  -  Marine  Corps  (GCSS-MC)  Increment  1 
FOT&ETest  Plan 

Global  Command  and  Control  System  -  Maritime  (GCCS-M)  Group  Level 
V4.1.0.1  IOT&E  Plan 

Global  Command  and  Control  System  -  Maritime  (GCCS-M)  Group  Level 
V4.1.0.1  IOT&E  Revised  Plan 

Global  Command  and  Control  System  -  Joint  (GCCS-J)  Version  4.3 
Update  1  OT&E  Plan 

GLOBALTFIUNDER  14  Information  Assurance  and  Interoperability 
Assessment  Plan 

Guided  Multiple  Launch  Rocket  System  (GMLRS)  -  Alternate  Warhead 
(AW)  Integrated  Developmental  Test/Operational  Test  (DT/OT)  OTA  Test 
Plan 

Guided  Multiple  Launch  Rocket  System  -  Alternative  Warhead 
(GMLRS-AW)  (IOT&E)  OTA  Test  Plan 

H-1  Upgrades  Program  AFI-1Z/UH-1Y  System  Configuration  Set  (SCS)  6.0 
Verification  of  Correction  of  Deficiency  (VCD)  Execution  DMAP 

Handheld,  Manpack,  Small  Form  Fit  -  Manpack  Radio  (HMS-MP)  FOT&E 
OTA  Test  Plan 

III  Marine  Expeditionary  Force  Exercise  Ulchi  Freedom  Guardian  (UFG)  14 
System  Assessment  Test  Plan 

Integrated  Defensive  Electronic  Counter  (IDECM)  Suite  Block  IV  FOT&ETest 
Plan 

Integrated  Electronic  Health  Record  (iEHR)  Increment  1  Single  Sign-on 
(SSO)/Context  Management  (CM)  OA  Plan 

Integrated  Electronic  Health  Record  (iEHR)  Increment  1  Single  Sign-On 
(SSO)/Context  Management  (CM)  Operational  Test  Plan 

Joint  Battle  Command  -  Platform  (JBC-P)  MOT&E  OTA  Test  Plan 

Joint  High  Speed  Vessel  (JHSV)  FOT&ETest  Plan  approval 

Joint  High  Speed  Vessel  (JHSV)  FOT&ETest  Plan  Change 

Joint  Inter  Agency  Task  Force  -  South  (JIATF-S)  Final  Assessment  Plan 

Joint  Light  Tactical  Vehicle  (JLTV)  Development  Test/Operational  Test 
(DT/OT)  OTA  Test  Plan 

Joint  Light  Tactical  Vehicle  (JLTV)  LUT  Operational  Test  Agency  Test  Plan 
Joint  Warning  and  Reporting  Network  (JWARN)  FOT&E  4  OTA  Test  Plan 

Joint  Warning  and  Reporting  Network  (JWARN)  Increment  1 
Modernization  FOT&E  2 Test  Plan 

Littoral  Combat  Ship  (LCS  3)  FREEDO/W-Variant  with  Surface  (SUW)  Mission 
Package  (MP)  Increment  2  Information  Assurance  (IA)  Revised  lOT&ETest 
Plan 

Littoral  Combat  Ship  (LCS  3)  FREEDO/W-Variant  with  Surface  Warfare  (SUW) 
Mission  Package  (MP)  Increment  2  Information  Assurance  (IA)  lOT&ETest 
Plan 


Littoral  Combat  Ship  (LCS)  with  Surface  Warfare  (SUW)  Mission  Package 
Increment  2  IOT&E  Plan 

Littoral  Combat  Ship  (LCS)  with  Surface  Warfare  (SUW)  Mission  Package 
Increment  2  IT  DMAP 

MH-60S  with  the  Airborne  Laser  Mine  Detection  Systems  (ALMDS)  and 
the  Airborne  Mine  Neutralization  System  (AMNS)  Phase  B  OATest  Plan 

MH-60R  Multi-mission  Helicopter  Digital  Rocket  Launcher  (DRL) 
Advanced  Precision  Kill  Weapon  System  II  (APKWS  II)  QRA  DMAP 

MH-60S  Digital  Rocket  Launcher  (DRL)  QRA  DMAP 

MH-60S  w/ Airborne  Mine  Neutralization  System  (AMNS)  OATest  Plan 

Miniature  Air  Launched  Decoy  (MALD)  Operational  Flight  Software 
(OFS)  8.0  FDE  Test  Plan 

Mk48  Mod  6  Advanced  Common  Torpedo  (ACOT)  and  Mk48  Mod  7 
Common  Broadband  Advanced  Sonar  System  (CBASS)  Torpedo  Test 
Plan  Addendum  for  Execution  of  Fleet  ICEX  per  Test  Plan  OT-CI  Change  2 

Mobile  Landing  Platform  (MLP)  Core  Capability  Set  (CCS)  IOT&E  Phase  I 
Test  Plan 

Nett  Warrior  IOT&E  Test  Plan 

P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA)  FOT&ETest  Plan 

P-8A  Poseidon  Multi-mission  Maritime  Aircraft  (MMA)  OT-C2  FOT&ETest 
Plan,  Change  1 

Patriot  Advanced  Capability -3  (PAC-3)  Missile  Segment  Enhancement 
(MSE)  Live  Fire  Test  (LFT)  OTA  Test  Plan 

Precision  Guidance  Kit  (PGK)  OTAOperational  Test  Plan 
QF-16  Full  Scale  Aerial  Target  (FSAT)  IOT&E  Plan 
Remote  Minehunting  System  (RMS)  IT  DMAP 
Ship  to  Shore  Connector  (SSC)  OA  Test  Plan 

Small  Tactical  Unmanned  Aircraft  System  (STUAS)  Joint 
COMOPTEVFOR/  MCOTEA  IOT&E  Test  Plan 

Standard  Missile-6  (SM-6)  OT-D1  FOT&ETest  Plan 

Surface  Electronic  Warfare  Improvement  Program  (SEWIP)  Block  2  IOT&E 
Test  Plan 

Surveillance  Towed  Array  Sensor  System/Low  Frequency  Active 
(SURTASSCLFA)  Test  Plan 

Tactical  Unmanned  Aircraft  System  Tactical  Common  Data  Link 
(Shadow)  FOT&E  OTA  Test  Plan 

Tempest  Wind  2014  (TW1 4)  Final  Assessment  Plan  (FAP) 

Turbo  Challenge  201 4  Information  Assurance  and  Interoperability 
Assessment  Plan 

United  States  Strategic  Command  Global  Lightening  14  Assessment 
Plan 

Valiant  Shield  201 4  Cybersecurity  and  Interoperability  Assessment  Plan 
Warfighter  Exercise  14-4  Final  Assessment  Plan  and  Addendum 
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LIVE  FIRE  TEST  AND  EVALUATION  STRATEGIES/MANAGEMENT  PLANS 

CVN  78  Gerald  R.  Ford  class  LFT&E  Management  Plan  USS  Fort  Worth  (LCS  3)  Total  Ship  Survivability  Trial  (TSST)  Plan 

Mobile  Landing  Platform  (MLP)  LFT&E  Management  Plan  VXX  Alternate  LFT&E  Strategy 

PATRIOT  Advanced  Capability-3  (PAC-3)  Missile  Segment  Enhancement 
(MS)  Program,  Detailed  LFT&E  Test  Plan  for  the  Penetration 
Characterization  for  6A1-4VTitanium  Fragments 


FY1 4  REPORTS  TO  CONGRESS 

PROGRAM 

DATE 

Initial  Operational  Test  and  Evaluation  Reports 

P-8  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA) 

October  201 3 

Joint  Battle  Command  -  Platform  (JBC-P)  with  classified  annex 

November  201 3 

MFI-60R  AN/APS-1 53  Automatic  Radar  Periscope  Detection  and  Discrimination  (ARPDD)  Program 

April  2014 

M982E1  Excalibur  Increment  1  b  with  classified  appendices 

June  2014 

Joint  High  Speed  Vessel  (JHSV) 

July  2014 

Multi-static  Active  Coherent  (MAC) 

July  2014 

Early  Fielding  Reports 

Massive  Ordnance  Penetrator  (MOP) 

October  201 3 

Joint  Tactical  Radio  System  (JTRS)  Flandheld,  Manpack,  Small  Form  Fit  (HMS)  Manpack  Radio 

December  201 3 

Littoral  Combat  Ship  (LCS) 

December  201 3 

Surveillance Towed-Array  Sensor  System  (SURTASS)  with  the  Compact  Low-Frequency  Active  (CLFA)  System 

February  2014 

Surface  ShipTorpedo  Defense  (SSTD) 

April  2014 

Follow-on  Operational  Test  and  Evaluation  Reports 

Virginia  class  Submarine  Dry  Deck  Shelter  (DDS) 

October  201 2 

MFI-60R/S  Flelicopters  Employing  AGM-1 1 4  HELLFIRE  Missiles  with  the  AN/AAS-44C(V)  Multi-Spectral  Targeting 
System  (MTS) 

April  2014 

USS  San  Antonio  LPD-1 7 

June  2014 

P-8A  Poseidon  Increment  1  Multi-Mission  Maritime  Aircraft 

June  2014 

Mk  54  Mod  0  Lightweight  Hybrid  Torpedo  with  Block  Upgrade  (BUG)  Tactical  Software 

September  201 4 

Live  Fire  Test  and  Evaluation  Reports 

Modernized  Expanded  Capacity  Vehicle  -  Survivability  (MECV-S) 

January  2014 

Operational  Test  and  Evaluation  Reports 

Cobra  King  (formerly  Cobra  Judy  Replacement  (CJR)) 

July  2014 

Operational  Assessment  Reports 

Nett  Warrior  at  Network  Integration  Evaluation  (NIE)  13.2  with  classified  annex 

November  201 3 

Special  Reports 

Response  to  the  House  Armed  Services  Committee  National  Defense  Authorization  Act  (HASC  NDAA)  Tasking  on 
Distributed  Common  Ground  System  -  Army  (DCGS-A) 

October  201 3 

Chief  of  Naval  Operations  Report  to  Congress  on  the  Current  Concept  of  Operations  and  Expected  Survivability 
Attributes  of  the  Littoral  Combat  Ship 

April  2014 

Ballistic  Missile  Defense  Reports 

FY1 3  Assessment  of  the  Ballistic  Missile  Defense  Systems  (includes  Classified  Appendices  A,  B,  C,  D,  and  E) 

February  2014 
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OTHER  FY1 4  REPORTS  (NOT  SENT  TO  CONGRESS) 

PROGRAM 

DATE 

Cybersecurity  Reports 

Defense  Connect  Online  (DCO)  Access  and  Authentication  Vulnerability  Assessment 

November  201 3 

U.S.  Special  Operations  Command  Exercise  EMERALD  WARRIOR  201 3 

February  2014 

U.S.  Army's  Warfighter  Exercise  1 3-4 

February  2014 

Electronic  Security  of  Special  Handling  Documents  Assessment 

April  2014 

Host  Based  Security  System  (HBSS)  Access  and  Authentication  Vulnerability  Assessment 

April  2014 

U.S.  Strategic  Command  Exercise  Global  Thunder  2014 

May  2014 

Aegis  Shipboard  Tactical  Data  Links  Finding  Memorandum 

June  2014 

FY1 3  Assessment  of  DOD  Information  Assurance  during  Major  Combatant  Command  and  Service  Exercises 

September  2014 

U.S.  Northern  Command  Exercise  Vigilant  Shield  2014 

September  2014 

U.S.Transportation  Command  ExerciseTurbo  Challenge  2014 

September  2014 

Early  Fielding  Reports 

Joint  Land  Attack  Cruise  Missile  Defense  (LACMD)  Elevated  Netted  Sensor  System  (JLENS) 

February  201 4 

Early  Operational  Assessment  Reports 

Ohio  Replacement 

July  2014 

Follow-on  Operational  Test  and  Evaluation  Reports 

Navy  Enterprise  Resource  Planning  (ERP) 

October  201 3 

C-1 7  Increased  Gross  Weight  (IGW)  and  Formation  Spacing  Reduction  (FSR) 

April  2014 

MV-22  OT-IIIJ 

June  2014 

Force  Development  Evaluation  Reports 

Global  Broadcast  Service  (GBS) 

November  201 3 

Air  Force  Distributed  Common  Ground  System  (AF  DCGS) 

August  2014 

MQ-9  Increment  One  Unmanned  Aircraft  System  (UAS)  Operational  Flight  Program  (OFP)  904.2  Software  Upgrade 

September  201 4 

Initial  Operational  Test  and  Evaluation  Reports 

Integrated  Personnel  and  Pay  System  -  Army  (IPPS-A)  Increment  1 

September  201 3 

Live  Fire  Test  and  Evaluation  Reports 

HELLFIRE  Romeo  Missile  Final  Lethality  Assessment 

May  2014 

Operational  Assessment  Reports 

Defense  Readiness  Reporting  System  -  Strategic  (DRRS-S) 

October  201 3 

Defense  Enterprise  Accounting  and  Management  System  (DEAMS) 

October  201 3 

USS  Gerald R.  Ford  (CVN  78)  class  Aircraft  Carrier 

December  201 3 

Remote  Minehunting  System  (RMS) 

January  2014 

Ground/AirTask  Oriented  Radar  (G/ATOR) 

January  2014 

Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM)  Basic  Electronic  Protection  Improvement  Program  (EPIP) 

May  2014 

Integrated  Electronic  Healthcare  Record  Increment  1  (iEHR)  Report 

May  2014 

Defense  Enterprise  Accounting  and  Management  System  (DEAMS)  Release  2  with  Classified  Appendix  B:  Cyber 
Economic  Vulnerability  Assessment  Report 

June  2014 

Mark  XXIIA  Mode  5  Joint  Operational  Test  Approach  (JOTA)  2 

July  2014 

AN/PRC-1 1 7G  with  classified  annex 

September  201 4 

NA28  Precision  Guidance  Kit  (PGK)  Urgent  Materiel  Release  (UMR)  Variant  Limited  User  Test  (LUT) 

September  201 4 

Operational  Test  and  Evaluation  Reports 

Theater  Medical  Information  Program  -  Joint  (TMIP-J) 

December  201 3 

Global  Command  and  Control  System  -  Joint  (GCCS-J)  Global  Release  4.3 

March  2014 
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OTHER  FY14  REPORTS  (NOT  SENTTO  CONGRESS)  (CONTINUED) 


PROGRAM 

DATE 

Special  Reports 

Joint  Land  Attack  Netted  Sensor  (JLENS)  Quick  Look  Assessment 

October  201 3 

Special  Operations  Forces  Mine  Resistant  Ambush  Protected  (MRAP)  -  All  Terrain  Vehicle  Underbody  Improvement 
Kit -Special  Operations  Command  (SOF  M-ATV  UIK-S)  Correction  of  Deficiencies 

November  201 3 

E-2D  Advanced  Flawkeye  (AHE)  Lot  2  Verification  of  Correction  of  Deficiencies  (VCD) 

March  2014 

Requirement  for  Use  of  a  Self-Defense  Test  Ship  (SDTS)  for  Operational  Testing  of  the  DDG  51  Flight  III  Equipped 
with  the  Air  and  Missile  Defense  Radar  (AMDR) 

September  2014 

Oversight  of  SubmarineTactical  Systems  (Interim  Operational  Testing  Assessment  for  the  Advanced  Process  Build 
201 1  (APB-1 1 )  Version  of  the  AN/BYG-1  (V)  Combat  Control  System  (CCS)  and  the  AN/BQQ-1 0  Acoustic  Rapid 
Commercial  Off-the-Shelf  (COTS)  Insertion  (A-RCI)  Sonar  System) 

September  2014 

Activity 
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Program  Oversight 


DOT&E  is  responsible  for  approving  the  adequacy  of  plans  for 
operational  test  and  evaluation  and  for  reporting  the  operational 
test  results  for  all  Major  Defense  Acquisition  Programs  to 
the  Secretary  of  Defense,  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  Service  Secretaries, 
and  Congress.  For  DOT&E  oversight  purposes,  Major  Defense 
Acquisition  Programs  were  defined  in  the  law  to  mean  those 
programs  meeting  the  criteria  for  reporting  under  Section  2430, 
Title  10,  United  States  Code  (USC)  (Selected  Acquisition 
Reports  (SARs)).  The  law  (sec. 139(a)(2)(B))  also  stipulates 
that  DOT&E  may  designate  any  other  programs  for  the  purpose 
of  oversight,  review,  and  reporting.  With  the  addition  of  such 
“non-major”  programs,  DOT&E  was  responsible  for  oversight  of 
a  total  of  309  acquisition  programs  during  FY 14. 

Non-major  programs  are  selected  for  DOT&E  oversight  after 
careful  consideration  of  the  relative  importance  of  the  individual 
program.  In  determining  non-SAR  systems  for  oversight, 
consideration  is  given  to  one  or  more  of  the  following  essential 
elements: 

•  Congress  or  OSD  agencies  have  expressed  a  high-level  of 
interest  in  the  program. 

•  Congress  has  directed  that  DOT&E  assess  or  report  on  the 
program  as  a  condition  for  progress  or  production. 

•  The  program  requires  joint  or  multi-Service  testing  (the  law 
(sec.  139(b)(4))  requires  DOT&E  to  coordinate  “testing 
conducted  jointly  by  more  than  one  military  department  or 
defense  agency”). 

•  The  program  exceeds  or  has  the  potential  to  exceed  the 
dollar  threshold  definition  of  a  major  program  according  to 
DOD  5000. 1,  but  does  not  appear  on  the  current  SAR  list 
(e.g.,  highly-classified  systems). 


•  The  program  has  a  close  relationship  to  or  is  a  key  component 
of  a  major  program. 

•  The  program  is  an  existing  system  undergoing  major 
modification. 

•  The  program  was  previously  a  SAR  program  and  operational 
testing  is  not  yet  complete. 

This  office  is  also  responsible  for  the  oversight  of  LFT&E 
programs,  in  accordance  with  10  USC  139.  DOD  regulation  uses 
the  term  “covered  system”  to  include  all  categories  of  systems 
or  programs  identified  in  10  USC  2366  as  requiring  LFT&E.  In 
addition,  systems  or  programs  that  do  not  have  acquisition  points 
referenced  in  10  USC  2366,  but  otherwise  meet  the  statutory 
criteria,  are  considered  “covered  systems”  for  the  purpose  of 
DOT&E  oversight. 

A  covered  system,  for  the  purpose  of  oversight  for  LFT&E, 
has  been  determined  by  DOT&E  to  meet  one  or  more  of  the 
following  criteria: 

•  A  major  system,  within  the  meaning  of  that  term  in 
10  USC  2302(5),  that  is: 

-  User-occupied  and  designed  to  provide  some  degree  of 
protection  to  the  system  or  its  occupants  in  combat 

-  A  conventional  munitions  program  or  missile  program 

•  A  conventional  munitions  program  for  which  more  than 
1,000,000  rounds  are  planned  to  be  acquired. 

•  A  modification  to  a  covered  system  that  is  likely  to  affect 
significantly  the  survivability  or  lethality  of  such  a  system. 

DOT&E  was  responsible  for  the  oversight  of  121  LFT&E 
acquisition  programs  during  FY14. 


Programs  Under  DOT&E  Oversight 
Fiscal  Year  2014 

(As  taken  from  the  September  2014  DOT&E  Oversight  List) 


DOD  PROGRAMS 


AC-130J 

BMDS  -  Ballistic  Missile  Defense  System  Program 
BMTC  -  Ballistic  Missile  Technical  Collection 

CHEM  DEMIL-ACWA  -  Chemical  Demilitarization  Program  -  Assembled 
Chemical  Weapons  Alternatives 

CHEM  DEMIL-CMA  -  Chemical  Demilitarization  (Chem  Demil)  -  Chemical 
Materals  Agency  (Army  Executing  Agent) 

Common  Analytical  Laboratory  System 


Defense  Agency  Initiative  (DAI) 

Defense  Enterprise  Accounting  and  Management  System  -  Increment  1 
(DEAMS-Inc.  1) 

Defense  Healthcare  Management  System  (DHMS) 

Defense  Medical  Information  Exchange  (DMIX) 

Defense  Readiness  Reporting  System  -  Strategic 

Defense  Security  Assistance  Management  System  (DSAMS)  -  Block  3 
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DOD  PROGRAMS  (continued) 


EDS  -  Explosive  Destruction  System 

Enterprise  Business  Accountability  System  -  Defense 

EProcurement 

Global  Combat  Support  System  -  Joint  (GCSS-J) 

Global  Command  and  Control  System  -  Joint  (GCCS-J) 

Joint  Aerial  Layer  Network 

Joint  Biological  Tactical  Detection  System 

Joint  Chemical  Agent  Detector  (JCAD) 

Joint  Command  and  Control  Capabilities  (JC2C)  [Encompasses  GCCS-FoS 
(GCCS-J,  GCCS-A,  GCCS-M,  TBMCS-FL,  DCAPES,  GCCS-AF,  USMC  JTCW, 
USMCTCO] 

Joint  Information  Environment 
Joint  Light  Tactical  Vehicle  (JLTV) 

Joint  Warning  and  Reporting  Network  (JWARN) 


Key  Management  Infrastructure  (KMI)  Increment  2 
MC-130J 

Mid-Tier  Networking  Vehicle  Radio 
milCloud 

Modernized  Intelligence  Database  (MIDB) 

Multi-Functional  Information  Distribution  System  (includes  integration 
into  USAF  and  USN  aircraft) 

Next  Generation  Chemical  Detector 
Next  Generation  Diagnostic  System 
Public  Key  Infrastructure  (PKI)  Incr  2 
SOCOM  Dry  Combat  Submersible  Medium  (DCSM) 

Teleport,  Generation  III 

Theater  Medical  Information  Program  -  Joint  (TMIP-J)  Block  2 


ARMY  PROGRAMS 


ABRAMSTANK  MODERNIZATION  -  Abrams Tank  Modernization  (M1E3) 
Abrams  Tank  Upgrade  (Ml  A1  SA/M1 A2  SEP) 

Advanced  Field  Artillery  Tactical  Data  System  (AFATDS)  Increment  2 
AFI-64E  Apache 

Airborne  and  Maritime/Fixed  Site  Joint  Tactical  Radio  System  (AMF  JTRS) 
Small  Airborne  Link  16Terminal  (SALT) 

Airborne  and  Maritime/Fixed  Site  Joint  Tactical  Radio  System  (AMF  JTRS) 
Small  Airborne  Networking  Radio  (SANR) 

AMF  Airborne  &  Maritime/Fixed  Station 
AN/PRC-1 17G  Radio 
AN/TPQ-53  Radar  System  (Q-53) 

Armed  Aerial  Scout  (previously  named  ARFI  Armed  Recon  Flelicopter) 
Armored  Multipurpose  Vehicle  (AMPV) 

Armored  Truck  -  Fleavy  Dump  Truck  (FHDT) 

Armored  Truck  -  Fleavy  Equipment  Transporter  (FHET) 

Armored  Truck  -  Fleavy  Expanded  Mobility  Tactical  Truck  (FIEMTT) 
Armored  Truck  -  M91 5A5  Line  Flauler 
Armored  Truck  -  M939  General  PurposeTruck 
Armored  Truck  -  Palletized  Loading  System  (PLS) 

Army  Vertical  Unmanned  Aircraft  System 

Biometrics  Enabling  Capability  (BEC)  Increment  1 

Biometrics  Enabling  Capability  Increment  0 

Black  FIAWK  (UFI-60L)  -  Utility  Flelicopter  Program 

Black  FIAWK  (UFI-60M)  -  Utility  Flelicopter  Program 

Bradley  Engineering  Change  Proposal  (ECP)  and  Modernization 

C-1 7  Increase  Gross  Weight  (IGW)  and  reduced  Formation  Spacing 
Requirements  (FSR)  withT-1 1  parachute 

Cartridge,  7.62mm,  M80A1 
CH-47F  -  Cargo  Flelicopter 


Common  Infrared  Countermeasures  (CIRCM) 

Common  Operating  Environment 
Common  Remotely  Operated  Weapons  System  III 
Department  of  Defense  Automated  Biometric  Information  System 
Distributed  Common  Ground  System  -  Army  (DCGS-A) 

EXCALIBUR-  Family  of  Precision,  155  mm  Projectiles 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and  Below  Program 

FBCB2  -  Joint  Capability  Release  (FBCB2  -  JCR) 

Field  Deployable  FHydrolysis  System  (FDFIS) 

Fixed-Wing  Utility  Aircraft 

FMTV  -  Family  of  Medium  Tactical  Vehicles 

General  Fund  Enterprise  Business  System  (GFEBS) 

Global  Combat  Support  System  -  Army  (GCSS-A) 

Guided  Multiple  Launch  Rocket  System  -  Unitary  (GMLRS  Unitary) 

Guided  Multiple  Launch  Rocket  System  -  Alternate  Warhead  (GMLRS  AW) 
HELLFIRE  Romeo 

High  Mobility  Multipurpose  Wheeled  Vehicle  (HMMWV) 

HIMARS  -  High  Mobility  Artillery  Rocket  System 

Identification  Friend  or  Foe  Mark  XIIA  Mode  5  (all  development  and 
integration  programs) 

Improved  Turbine  Engine  Program 

Indirect  Fire  Protection  Capability  Increment  2  -  Intercept 

Integrated  Air  and  Missile  Defense  (IAMD) 

Integrated  Personnel  and  Pay  System  -  Army  (Army  IPPS)  Increment  1 

Integrated  Personnel  and  Pay  System  -  Army  (IPPS-A)  Increment  2 

Interceptor  Body  Armor 

Javelin  Antitank  Missile  System  -  Medium 

Joint  Air-to-Ground  Missile 
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ARMY  PROGRAMS  (continued) 


Joint  Assault  Bridge 

Joint  Battle  Command  -  Platform  (JBC-P) 

Joint  Future  Theater  Lift  Concept  (JFTLC) 

Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System 
Joint  Personnel  Identification  (JPIv2) 

Joint  Tactical  Network  (JTN) 

Kiowa  Warrior,  OH-58F  Cockpit  and  Sensor  Upgrade  Program  (CASUP) 
Logistics  Modernization  Program  (LMP) 

Long  Range  Precision  Fires  (LRPF) 

Ml  200  Knight  Targeting  Under  Armor  (TUA) 

M270A1  Multiple  Launch  Rocket  System  (MLRS) 

M829E4 

Modernized  Expanded  Capacity  Vehicle  (MECV)  -  Survivability  Project 
MQ-1 C  Unmanned  Aircraft  System  Gray  Eagle 
Nett  Warrior 

One  System  Remote  Video  Terminal 
Paladin/FASSV  Integrated  Management  (PIM) 

PATRIOT  PAC-3  -  Patriot  Advanced  Capability  3  (Missile  only) 

PATRIOT/MEADS  -  Patriot/Medium  Extended  Air  Defense  System 

RQ-1 1 B  Raven  -  Small  Unmanned  Aircraft  System 

RQ-7B  SFIADOW  -Tactical  Unmanned  Aircraft  System 

Soldier  Protection  System 

Spider  XM7  Network  Command  Munition 

Stryker  ECP  -  Stryker  Engineering  Change  Proposal 

Stryker  Ml  126  Infantry  Carrier  Vehicle  including  Double  V-Hull  variant 


Stryker  Ml  127  Reconnaissance  Vehicle 
Stryker  Ml  1 28  Mobile  Gun  System 

Stryker  Ml  129  Mortar  Carrier  including  the  Double  V-Hull  variant 
Stryker  Ml  130  Commander's  Vehicle  including  the  Double  V-Hull  Variant 
Stryker  Ml  131  Fire  Support  Vehicle  Including  the  Double  V-Hull  Variant 

Stryker  Ml  132  Engineer  Squad  Vehicle  Including  the  Double  V-Hull 
Variant 

Stryker  Ml  133  Medical  Evacuation  Vehicle  Including  the  Double  V-Hull 
Variant 

Stryker  Ml  1 34  ATGM  Vehicle  Including  the  Double  V-Hull  Variant 
Stryker  Ml  135  NBC  Reconnaissance  Vehicle  (NBCRV) 

Tactical  Mission  Command 
Tactical  Radio  System  Manpack 
Tactical  Radio  System  Rifleman  Radio 
UH-72A  Lakota  Light  Utility  Helicopter 

WIN-T  INCREMENT  1  -  Warfighter  Information  Network  -  Tactical 
Increment  1 

WIN-T  INCREMENT  2 -Warfighter  Information  Network  -  Tactical 
Increment  2 

WIN-T  INCREMENT  3 -Warfighter  Information  Network  -  Tactical 
Increments 

WIN-T  INCREMENT  4 -Warfighter  Information  Network  -  Tactical 
Increment  4 

XM1 156  Precision  Guidance  Kit  (PGK) 

XM25,  Counter  Defilade  Target  Engagement  (CDTE)  System 
XM395  Accelerated  Precision  Mortar  Initiative  (APMI) 


NAVY  PROGRAMS 


Advanced  Airborne  Sensor 

Advanced  Extremely  High  Frequency  Navy  Multiband  Terminal  Satellite 
Program  (NMT) 

AEGIS  Modernization  (Baseline  Upgrades) 

AGM-88E  Advanced  Anti-Radiation  Guided  Missile 
AH-1Z 

AIM-9X  -  Air-to-Air  Missile  Upgrade  Block  II 
Air  and  Missile  Defense  Radar  (AMDR) 

Air  Warfare  Ship  Self  Defense  Enterprise 

Airborne  Laser  Mine  Detection  System  (AN/AES-1 )  (ALMDS) 

Airborne  Mine  Neutralization  System  (AN/ASW-235)  (AMNS) 

Airborne  Resupply/Logistics  for  Seabasing 
Amphibious  Assault  Vehicle  Upgrade 
Amphibious  Combat  Vehicle  (ACV) 

AN/APR-39  Radar  Warning  Receiver 

AN/AQS-20A  Minehunting  Sonar 

AN/SQQ-89A(V)  Integrated  USW  Combat  Systems  Suite 


Assault  Breaching  System  Coastal  Battlefield  Reconnaissance  and  Analysis 
System  Block  I 

Assault  Breaching  System  Coastal  Battlefield  Reconnaissance  and  Analysis 
System  Block  II 

CH-53K-  Heavy  Lift  Replacement  Program 
Close-In  Weapon  System  (CIWS)  including  SEARAM 
Cobra  Judy  Replacement  -  Ship-based  radar  system 
Common  Aviation  Command  and  Control  System  (CAC2S) 

Consolidated  Afloat  Networks  and  Enterprise  Services  (CANES) 
Cooperative  Engagement  Capability  (CEC) 

Countermeasure  Anti-Torpedo 

CVN-78  -  Gerald  R.  Ford  class  Nuclear  Aircraft  Carrier 

DDG  1 000  -Zumwalt  class  Destroyer  -  includes  all  supporting  PARMsand 
the  lethality  of  the  LRLAP  and  30mm  ammunition 

DDG  51  -Arleigh  Burke  class  Guided  Missile  Destroyer  -  includes  all 
supporting  PARMs 

DDG  51  Flight  III  -  Arleigh  Burke  class  Guided  Missile  Destroyer  -  includes 
all  supporting  PARMs 
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NAVY  PROGRAMS  (continued) 


Dept  of  Navy  Large  Aircraft  Infrared  Countermeasures  Program 
Distributed  Common  Ground  System  -  Navy  (DCGS-N) 

Distributed  Common  Ground  System  -  Marine  Corps  (DCGS-MC) 

E-2D  Advanced  Hawkeye 

EA-1 8G  -  Airborne  Electronic  Attack 

Electro-Magnetic  Aircraft  Launching  System 

Enhanced  Combat  Helmet 

Evolved  Sea  Sparrow  Missile  (ESSM) 

Evolved  Sea  Sparrow  Missile  Block  2 

F/A-1 8E/F  -  SUPER  HORNET  Naval  Strike  Fighter 

Future  Pay  and  Personnel  Management  Solution  (FPPS) 

Global  Combat  Support  System  -  Marine  Corps  (GCSS-MC) 

Global  Command  and  Control  System  -  Maritime  (GCCS-M) 
Ground/AirTask Oriented  Radar  (G/ATOR) 

Identification  Friend  or  Foe  Mark  XIIA  Mode  5  (all  development  and 
integration  programs) 

Infrared  Search  and  Track  System 

Integrated  Defensive  Electronic  Countermeasures  Block 4 
Joint  and  Allied  Threat  Awareness  System 
Joint  High  Speed  Vessel  (JHSV) 

JOINT  MRAP  -  Joint  Mine  Resistant  Ambush  Protected  Vehicles 
FOV  -  including  SOCOM  vehicles 

Joint  Precision  Approach  and  Landing  System  Increment  1  (Ship  system) 
Joint  Precision  Approach  and  Landing  System  Increment  2  (Land  system) 
Joint  Stand-Off  Weapon  C-1  variant  (JSOW  C-1 ) 

KC-1 30J  with  Harvest  Hawk 
Landing  Ship  Dock  Replacement  (LX(R)) 

Large  Displacement  Unmanned  Undersea  Vehicle 
LCS  Interim  Surface  to  Surface  Missile 

LHA  6  -AMERICA  class  -  Amphibious  Assault  Ship  -  includes  all 
supporting  PARMs 

LHA  8  Amphibious  Assault  Ship  ( America  class  with  well  deck) 

Light  Armored  Vehicle 

Light  Weight  Tow  Torpedo  Countermeasure  (part  of  LCS  ASW  Mission 
Module) 

Littoral  Combat  Ship  (LCS)  -  includes  all  supporting  PARMs,  and  57mm 
lethality 

Littoral  Combat  Ship  Mission  Modules  including  30mm 

Littoral  Combat  Ship  Surface-to-Surface  Missile  (follow  on  to  the  interim  SSM) 

Littoral  Combat  Ship  Variable  Depth  Sonar  (LCS  VDS) 

Logistics  Vehicle  System  Replacement 

LPD  1 7  -  SAN  ANTONIO  class  -  Amphibious  Transport  Dock  Ship  -  includes 
all  supporting  PARMs  and  30  mm  lethality 

Marine  Personnel  Carrier 

Medium  Tactical  Vehicle  Replacement  Program  (USMC)  (MTVR) 

MH-60R  Multi-Mission  Helicopter  Upgrade 
MH-60S  Multi-Mission  Combat  Support  Helicopter 
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Mk  54  torpedo/MK  54  VLA/MK  54  Upgrades  Including  High  Altitude  ASW 
Weapon  Capability  (HAAWC) 

MK-48  CBASS Torpedo  including  all  upgrades 

Mobile  Landing  Platform  (MLP)  Core  Capability  Set  (CCS)  Variant  and  MLP 
Afloat  Forward  Staging  Base  (AFSB)  Variant 

Mobile  User  Objective  System  (MUOS) 

MQ-4C  Triton 

MQ-8- Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle  (VTUAV) 
(Fire  Scout) 

Multi-static  Active  Coherent  (MAC)  System  CNO  project  1 758 
Naval  Integrated  Fire  Control  -  Counter  Air  (NIFC-CA)  From  the  Air 
Naval  Integrated  Fire  Control  -  Counter  Air  (NIFC-CA)  From  the  Sea 
Navy  Enterprise  Resource  Planning  (ERP) 

Next  Generation  Jammer 

Offensive  Anti-Surface  Warfare  Increment  1 

Offensive  Anti-Surface  Warfare,  Increment  2 

Ohio  Replacement  Program  (Sea-based  Strategic  Deterrence)  -  including 
all  supporting  PARMs 

OSPREY  MV-22  -  Joint  Advanced  Vertical  Lift  Aircraft 

P-8A  Poseidon  Program 

Remote  Minehunting  System  (RMS) 

Replacement  Oiler 

Rolling  Airframe  Missile  (RAM)  including  RAM  Block  1 A  Helicopter  Aircraft 
Surface  (HAS)  and  RAM  Block  2  Programs 

Ship  Self  Defense  System  (SSDS) 

Ship  to  Shore  Connector 

Small  Tactical  Unmanned  Aerial  System  (STUAS)  -  UASTier  II 

SSN  774  VIRGINIA  class  Submarine 

SSN  784  VIRGINIA  class  Block  III  Submarine 

Standard  Missile-2  (SM-2)  including  all  mods 

Standard  Missile-6  (SM-6) 

Submarine  Torpedo  Defense  System  (SubTDS)  including 
countermeasures  and  Next  Generation  Countermeasure  System  (NGCM) 

Surface  Electronic  Warfare  Improvement  Program  (SEWIP)  Block  2 
Surface  Electronic  Warfare  Improvement  Program  (SEWIP)  Block  3 

Surface  Mine  Countermeasures  Unmanned  Undersea  Vehicle  (also  called 
Knifefish  UUV)  (SMCM  UUV) 

Surveillance  Towed  Array  Sonar  System/Low  Frequency  Active 
(SURTASS/LFA)  including  Compact  LFA  (CLFA) 

Torpedo  Warning  System  (Previously  included  with  Surface  ShipTorpedo 
Defense  System)  including  all  sensors  and  decision  tools 

TRIDENT  II  MISSILE  -  Sea  Launched  Ballistic  Missile 
UH-1Y 

Unmanned  Carrier  Launched  Airborne  Surveillance  and  Strike  System 

Unmanned  Influence  Sweep  System  (UISS)  include  Unmanned  Surface 
Vessel  (USV)  and  Unmanned  Surface  Sweep  System  (US3) 

USMC  MRAP -Cougar 

VXX  -  Presidential  Helicopter  Fleet  Replacement  Program 
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AIR  FORCE  PROGRAMS 


20mm  PGU-28/B  Replacement  Combat  Round 
Advanced  Pilot  Trainer 

AEHF  -  Advanced  Extremely  High  Frequency  (AEHF)  Satellite  Program 

AFNet  Modernization  capabilities  (Bitlocker,  Data  at  Rest  (DaR),  Situational 
Awareness  Modernization  (SAMP)) 

AFNET  Vulnerability  Management  (AFVM)  -  Assured  Compliance 
Assessment  Solution  (ACAS) 

AIM-1 20  Advanced  Medium-Range  Air-to-Air  Missile 
Air  Force  Distributed  Common  Ground  System  (AF-DCGS) 

Air  Force  Integrated  Personnel  and  Pay  System  (AF-IPPS) 

Air  Operations  Center  -  Weapon  System  (AOC-WS)  initiatives  including 
10.0  and  10.1 

Air  Operations  Center -Weapon  System  (AOC-WS)  initiative  10.2 
Airborne  Signals  Intelligence  Payload  (ASIP)  Family  of  Sensors 

Airborne  Warning  and  Control  System  Block  40/45  Computer  and  Display 
Upgrade 

ALR-69A  Radar  Warning  Receiver 

B-2  Defensive  Management  System  Modernization  (DMS) 

B-2  Extremely  High  Frequency  SATCOM  and  Computer  Increment  1 

B-2  Extremely  High  Frequency  SATCOM  and  Computer  Increment  2 

B61  Mod  12  Life  Extension  Program 

Base  Information  Transport  Infrastructure  (BITI)  -  Wireless 

Battle  Control  System  -  Fixed  (BCS-F)  3.2 

C-1 30J  -  HERCULES  Cargo  Aircraft  Program 

C-5  Aircraft  Reliability  Enhancement  and  Re-engining  Program 

C-5  Core  Mission  Computer  and  Weather  Radar  Replacement 

Cobra  Judy  Replacement  Mission  Planning  Tool 

Combat  Rescue  Helicopter  (CRH) 

Command  and  Control  Air  Operations  Suite  (C2AOS)/Command 
and  Control  Information  Services  (C2IS)  (Follow-on  to  Theater  Battle 
Management  Core  Systems) 

Defense  Enterprise  Accounting  and  Management  System  -  Air  Force 
(DEAMS  -  AF) 

ECSS  -  Expeditionary  Combat  Support  system 
Enclave  Control  Node  (ECN) 

EPS  -  Enhanced  Polar  System 

F-15  Eagle  Passive  Active  Warning  Survivability  System 
F-15E  Radar  Modernization  Program 
F-22  -  RAPTOR  Advanced  Tactical  Fighter 


F-35  -  Lightning  II  Joint  Strike  Fighter  (JSF)  Program 

FAB-T  -  Family  of  beyond  Line-of-Sight  Terminals 

Full  Scale  Aerial  Target 

GBS  -  Global  Broadcast  Service 

Geosynchronous  Space  Situational  Awareness  Program 

GPS  OCX  -  Global  Positioning  Satellite  Next  Generation  Control  Segment 

GPS-IIIA  -  Global  Positioning  Satellite  III 

Hard  Target  Munition 

HC/MC-1 30  Recapitalization 

Identification  Friend  or  Foe  Mark  XIIA  Mode  5  (all  development  and 
integration  programs) 

Integrated  Strategic  Planning  and  Analysis  Network  (ISPAN)  Increment  2 
Integrated  Strategic  Planning  and  Analysis  Network  (ISPAN)  Increment  4 
Joint  Air-to-Surface  Standoff  Missile  Extended  Range 
Joint  Space  Operations  Center  Mission  System  (JMS) 

JSTARS  Recapitalization 
KC-46  -Tanker  Replacement  Program 
Long  Range  Stand  Off  (LRSO)  Weapon 
Long  Range  Strike  Bomber 
Massive  Ordnance  Penetrator  (MOP) 

Military  GPS  User  Equipment  (GPS  MGUE) 

Miniature  Air  Launched  Decoy  -  Jammer  (MALD-J) 

MQ-9  REAPER  -  Unmanned  Aircraft  System 
Multi-Platform  RadarTechnology  Insertion  Program 

NAVSTAR  Global  Positioning  System  (GPS)  (includes  Satellites,  Control  and 
User  Equipment) 

OSPREY  CV-22  -  Joint  Advanced  Vertical  Lift  Aircraft 
Presidential  Aircraft  Recapitalization 

RQ-4B  Block  30  -  High  Altitude  Endurance  Unmanned  Aircraft  System 

RQ-4B  Block 40  Global  Hawk  -  High  Altitude  Long  Endurance  Unmanned 
Aircraft  System 

SBIRS  HIGH  -  Space-Based  Infrared  System  Program,  High  Component 

SBSS  B10  Follow-on  -  Space-Based  Space  Surveillance  Block  1 0  Follow-on 

SF  -  Space  Fence 

SIPRNET  Modernization 

Small  Diameter  Bomb,  Increment  II 

Three-Dimensional  Expeditionary  Long-Range  Radar 

Weather  Satellite  Follow-on  (WSF) 


Program  Oversight 
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Problem  Discovery  Affecting  OT&E 


Background 

In  2011,  Congress  expressed  concern  that  acquisition  programs  are  discovering  problems  during  operational  testing  (OT) 
that:  (1)  should  have  been  discovered  in  developmental  testing  (DT),  and  (2)  should  have  been  corrected  prior  to  OT.  In 
response  to  this  congressional  concern,  I  added  this  section  to  my  Annual  Report,  as  a  means  to  survey  across  all  DOT&E 
oversight  programs  covered  in  this  report,  the  extent  of  late  problem  discovery  and  to  identify  known  problems  that 
jeopardize  a  system’s  successful  performance  in  upcoming  OT. 

This  is  the  fourth  time  this  section  has  been  included  in  my  Annual  Report,  and  this  iteration  presents  a  more  in-depth  review 
of  the  programs  included  in  this  report.  Last  year,  this  section  consisted  of  short  case  studies  that  discussed  problems  that 
were  identified  during  OT  or  DT.  This  year’s  section  includes  data  that  break  down  into  several  relevant  categories  the 
effectiveness,  suitability,  and  cybersecurity  problems  that  were  either  observed  during  OT  or  that  jeopardize  a  system’s 
successful  performance  in  an  upcoming  OT  event  (i.e.,  if  known  problems  are  not  fixed,  a  finding  of  not  effective,  not 
suitable,  and/or  not  survivable  could  occur).1  The  results  presented  in  this  section  continue  to  show  that  OT  is  necessary,  and 
that  we  continue  to  find  significant  and  substantial  problems  during  OT  that  were  either  not  previously  observed  or  could  not 
be  observed  in  DT.  Also,  as  documented  in  this  section,  OT  continues  to  identify  problems  that  were  previously  discovered 
but  not  fixed. 

Overview  of  Problem  Discovery  in  OT 

Figure  1  below  shows  a  breakdown 
of  the  number  of  significant  problems 
(per  program  and  by  the  phase  of 
testing)  and  where  the  problems  were 
newly  discovered  or  already  known. 

As  expected,  the  rate  of  new  problem 
discovery  in  early  OT  that  occurs 
prior  to  Initial  Operational  Test  and 
Evaluation  (IOT&E)  (pre-OT  conducted 
to  inform  acquisition  and/or  early 
fielding  decisions)  is  higher  than  the  rate 
of  problem  discovery  in  both  IOT&E 
and  Follow-on  Operational  Test  and 
Evaluation  (FOT&E).  This  is  a  desirable 
trend  because  the  earlier  a  problem  is 
discovered,  the  easier  it  is  to  fix  it,  and  it 
is  consistent  with  DOT&E ’s  initiative  for 
early  involvement  in  test  programs.  The 
ratio  for  new  problem  discovery  (black 
bars)  is  the  highest  (two  “significant 
problems”  per  program)  for  early  OT. 

Significant  problems  are  those  that  would  have  a  negative  impact  on  DOT&E’s  assessment  of  effectiveness,  suitability,  or 
cybersecurity. 

For  re-observations  of  known  problems,  the  rate  is  also  higher  in  early  OT,  and  is  higher  overall  than  for  new  problem 
discovery  (red  bars).  This  result  indicates  that  while  early  OT  is  effective  in  demonstrating  the  operational  impact  of 
known  problems  prior  to  IOT&E,  OT  is  observing  more  known  problems  in  all  phases  of  testing  compared  to  new  problem 
discovery.2  In  cases  where  known  issues  prior  to  OT  are  significant  (indicating  a  lack  of  system  maturity),  DOT&E  has 
suggested  not  doing  the  OT  because  the  resources  expended  conducting  the  test  would  not  be  worth  the  little  or  irrelevant 
information  gained  from  an  OT  at  that  time.  This  year,  DOT&E  suggested  foregoing  planned  OT  events  for  the  F-35  Joint 
Strike  Fighter  (JSF),  Air  and  Missile  Defense  Radar  (AMDR),  and  Remote  Minehunting  System  (RMS)  because  of  several 
known  performance  issues. 

1 .  Cybersecurity  problems  are  evaluated  through  OT  and  are  considered  in  DOT&E’s  survivability  assessments.  Survivability  problems 
discovered  through  Live  Fire  Test  and  Evaluation  are  not  included  in  this  discussion  of  OT. 

2.  For  pre-IOT&E  testing,  observing  known  problems  is  not  a  major  issue  because  the  program  still  has  time  to  correct  them  prior  to  IOT&E;  this 
fact  underscores  the  importance  of  conducting  an  operational  assessment  prior  to  the  Milestone  C  or  Low -Rate  Initial  Production  decision. 
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New  Problem  Discovery 
Known  Problem  Re-observations 


Pre-IOT&E 


IOT&E 

Phase  of  Testing 


FOT&E 


Pre-IOT&E 

IOT&E 

FOT&E 

New  Problem 
Discovery 

Total  Problems 

10 

14 

8 

Total  Programs 

5 

8 

5 

Ratio 

2 

1.75 

1.6 

Known  Problem 

Re-observations 

Total  Problems 

13 

19 

14 

Total  Programs 

5 

10 

10 

Ratio 

2.6 

1.9 

1.4 

FIGURE  1.  PROBLEM  DISCOVERY  RATIOS  IN  PRE-IOT&E,  IOT&E,  AND  FOT&E 
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Other  trends  are: 

•  About  one-third  of  the  programs  that  underwent  OT  during  FY14  did  so  successfully;  that  is,  they  did  not  uncover 
problems  significant  enough  to  negatively  affect  my  assessment  of  operational  effectiveness,  suitability,  or  cybersecurity. 

•  For  new  problem  discovery,  about  half  of  the  effectiveness  problems  found  in  OT  were  not  discoverable  in  DT  because  the 
operationally  realistic  conditions  required  in  OT  were  needed  to  discover  the  problem  (i.e.,  testing  under  realistic  combat 
conditions  by  typical  military  users). 

•  More  than  two-thirds  of  the  programs  that  commenced  IOT&E  or  FOT&E  in  FY14  with  known  suitability  problems 
implemented  (and  in  many  cases  tested)  fixes  to  these  problems  prior  to  the  OT.  This  is  an  area  where  DOT&E’s 
initiatives  on  reliability  growth  are  having  a  positive  effect. 

•  Looking  to  the  future  and  consistent  with  the  first  bullet  above,  about  one-third  of  the  programs  with  upcoming  OT  events 
in  the  next  three  years  have  not  yet  exhibited  any  effectiveness,  suitability,  or  cybersecurity  problems  significant  enough 
to  jeopardize  successful  performance  in  OT.  (However,  we  know  that  about  half  of  the  new  problems  observed  during  OT 
cannot  be  observed  in  early  testing  because  of  the  need  for  operationally  realistic  environments.) 

•  Thirty  percent  of  the  programs  undergoing  OT  in  FY14  only  re-observed  previous  known  problems  during  OT;  no 
additional  significant  problems  were  found. 

•  Pre-IOT&E  test  events  are  more  likely  to  be  delayed  to  allow  time  to  correct  problems  compared  to  delaying  either 
IOT&Es  or  FOT&Es. 

•  A  majority  of  programs  (10/13)  that  observed  problems  during  IOT&E  re -observed  at  least  one  problem  that  was  known 
prior  to  the  IOT&E. 

•  Five  of  the  nine  programs  that  re -observed  known  effectiveness  issues  during  an  IOT&E  or  FOT&E  in  FY14  did  not 
identify  fixes  to  address  these  problems  prior  to  operational  test. 


EVALUATION  OF  PROBLEM  DISCOVERY 


Programs  with  an  FY140T 

I  surveyed  81  programs  that  either  underwent  OT  in  FY14  or  will  undergo  OT  within  the  next  three  years  (some  programs 
fall  into  both  categories),  and  are  reported  on  in  this  Annual  Report.3  The  results  presented  in  this  section,  including  those  in 
Figure  1  above,  focus  on  these  programs.  I  classified  the  programs  that  underwent  OT  into  one  of  three  main  categories:  (1) 
successful  performance  in  OT;  (2)  new  performance  problems  discovered;  and  (3)  known  performance  problems  re-observed. 
The  more  detailed  review  conducted  this 
year  also  allowed  me  to  categorize  individual 
problems  and  Program  Office  responses  to 
these  problems,  whereas  last  year’s  report 
only  categorized  problems  at  the  program 
level.  Otherwise,  the  categories  used  in 
this  year’s  report  are  similar  to  those  used 
in  previous  years.  These  categories  are 
described  in  Table  1.  For  problems  that  were 
discovered  during  OT,  I  assess  whether  these 
problems  affected  effectiveness,  suitability, 
or  cybersecurity  and  whether  they  reasonably 
could  have  been  discovered  prior  to  the  OT 
event. 

Programs  with  an  upcoming  OT 

For  programs  that  are  scheduled  to  undergo 
an  OT  event  within  the  next  three  years,  I 
identified  those  that  have  not  uncovered  any  problems  that  jeopardize  a  system’s  successful  performance  in  upcoming  OT 
events,  and  those  with  problems  significant  enough  that,  if  uncorrected,  would  negatively  affect  my  assessment  of  operational 
effectiveness,  suitability,  and/or  cybersecurity.  I  classify  these  programs  into  one  of  three  categories:  (1)  no  problems  for 
upcoming  OT;  (2)  problems  delayed  upcoming  OT;  and  (3)  problems  have  not  delayed  upcoming  OT.  They  are  described  in 
more  detail  in  Table  2. 


TABLE  1 .  PROBLEM  DISCOVERY  CATEGORIES  FOR  PROGRAMS 

WITH  AN  OT  EVENT  OR  A  DOT&E  REPORT  IN  THE  LAST  YEAR 

Category 

Description 

Successful  OT 

Successful  OT 
without  delays 

No  significant  problems  were  discovered  during  OT  that 
would  negatively  affect  DOT&E's  assessment  of  operational 
effectiveness,  suitability,  and/or  cybersecurity. 

Successful,  but 
delayed,  OT 

Problem(s)  were  discovered  that  delayed  entry  into  OT  so 
they  could  be  addressed,  thus  contributing  to  a  successful  OT 
outcome. 

New  problem  discovery 

Problem(s)  significant  enough  to  negatively  affect  DOT&E's 
assessment  of  operational  effectiveness,  suitability,  and/or 
cybersecurity  were  discovered  for  the  first  time  in  OT. 

Known  problem 
re-observations 

Problem(s)  were  observed  in  OT  that  were  known  prior  to 
entering  OT  and  significant  enough  to  negatively  affect  DOT&E's 
assessment  of  operational  effectiveness,  suitability,  and/or 
cybersecurity. 

3.  The  original  congressional  request  specified  programs  scheduled  to  commence  operational  testing  within  the  next  two  years.  I  expanded  that 
window  to  three  years  to  include  programs  that  delayed  their  entry  into  OT  so  they  could  fix  known  problems  before  commencing  OT. 
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TABLE  2.  PROBLEM  DISCOVERY  IN  CATEGORIES  FOR  PROGRAMS  IN  THIS  ANNUAL  REPORT 
SCHEDULEDTO  UNDERGO  OT  WITHIN  THE  NEXT  3  YEARS 

Category 

Description 

No  problems  for 
upcoming  OT 

The  program  has  not  exhibited  any  problems  that  would  negatively  affect  DOT&E's 
assessment  of  operational  effectiveness,  suitability,  and/or  cybersecurity  for  the 
upcoming  OT.1 

Problems  delayed 
upcoming  OT 

Problems  exist  that,  if  not  corrected,  would  negatively  affect  DOT&E's  assessment  of 
operational  effectiveness,  suitability,  and/or  cybersecurity  for  the  upcoming  OT.  OT 
has  been  delayed  so  these  problems  can  be  addressed  prior  to  commencing  OT. 

Problems  have  not 
delayed  upcoming  OT 

Problems  exist  that,  if  not  corrected,  would  negatively  affect  DOT&E's  assessment  of 
operational  effectiveness,  suitability,  and/or  cybersecurity  for  the  upcoming  OT,  but 

OT  has  not  (yet)  been  delayed  to  address  these  problems. 

1 .  Such  programs  could  be  examples  where  the  program  development  process,  including  DT  and  OT,  appears  to  be  moving 
along  well.  Alternatively,  the  testing  to  date  might  not  have  been  sufficiently  stressing  to  surface  any  problems. 

SUMMARY  OF  RESULTS 


For  problem  discovery  in  FY14, 1  found  a  mixture  of  positive  trends  and  areas  that  may  need  improvement.  The  results 
are  shown  in  Figure  2.  Blocks  that  are  colored  green  signify  positive  trends,  while  the  block  in  red  signifies  areas  that  need 
improvement.  The  yellow  block  represents  an  outcome  that  is  in-between  or  neutral.  The  two  blocks  with  a  yellow/green 
color  gradient  are  a  combination  of  mixed  results.  The  outcomes  shown  in  Figure  2  are  discussed  in  more  detail  in  the 
sections  that  follow. 


About  one-third  of  the  programs 
(15  of  48)  that  underwent  OT  during 
FY14  did  so  successfully;  that  is, 
they  did  not  uncover  problems 
significant  enough  to  negatively 
affect  DOT&E ’s  assessment  of 
operational  effectiveness,  suitability, 
and  cybersecurity.4  About 
two-thirds  of  the  programs  (33  of 
48)  that  underwent  OT  during  FY14 
encountered  problems  that  negatively 
affected  DOT&E ’s  assessment  of  their 
operational  effectiveness,  suitability, 
and/or  cybersecurity.  These 
problems  were  either  new  problems 
discovered  in  the  OT  event,  or  were 
re-observations  of  known  problems. 
Of  these  33  programs,  8  programs 
discovered  only  new  problems, 

15  only  re-observed  known  problems, 
and  10  both  discovered  new  problems 
and  re-observed  known  problems. 
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'Some  programs  both  underwent  OT  in  FY14  and  have  an 
upcoming  OT  within  the  next  three  years,  so  the  sum  of 
these  programs  (48  +  42  =  90)  exceeds  the  actual  number 
of  programs  surveyed  (81). 
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FIGURE  2.  PROBLEM  DISCOVERY  IN  OT  RESULTS  MEASURED  BY  PROGRAM  COUNTS 


For  programs  with  upcoming  OT  events  in  the  next  three  years,  I  determined  that  slightly  more  than  one-third  (15  of  42)  of 
the  programs  currently  exhibit  no  effectiveness,  suitability,  or  cybersecurity  problems  significant  enough  to  jeopardize  their 
successful  performance  in  upcoming  OT,  which  is  to  say  that  no  problems  have  yet  been  found  that,  if  not  corrected,  would 
negatively  affect  my  assessment  of  operational  effectiveness,  suitability,  or  cybersecurity.  Of  the  remaining  two-thirds  of 
programs  (27),  I  identified  23  that  have  effectiveness,  suitability,  and/or  cybersecurity  problems  that,  if  not  corrected,  could 
negatively  affect  my  operational  assessments.  The  remaining  four  programs  have  items  that  potentially  jeopardize  successful 
performance  in  OT,  but  these  relate  more  to  schedule  or  process  as  opposed  to  effectiveness,  suitability,  or  cybersecurity. 
Examples  include  test  schedules  in  Test  and  Evaluation  Master  Plans  that  are  not  executable;  reliance  on  other  programs  that 
are  facing  development  challenges;  and  failed,  cancelled,  or  delayed  DTs  that  jeopardize  successful  performance  in  OT. 


4.  Note  that  even  in  these  cases,  OT  provides  recommendations  or  potential  improvements  to  improve  system  perfonnance  for  the  warfighter. 
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PROGRAMS  THAT  CONDUCTED  OT  IN  FY14 

For  programs  that  conducted  an  OT  event  in  FY14,  my  analysis  consists  of  the  following: 

•  Program-Level  -  Analysis  of  the  number  of  programs  that  fall  into  each  of  the  categories  listed  in  Table  1,  broken  down  by 
the  types  of  problems  found  by  each  program  (effectiveness,  suitability,  or  cybersecurity),  and  across  the  three  phases  of 
OT  (pre-IOT&E,  IOT&E,  or  FOT&E) 

•  Problem-Level  -  For  programs  that  experienced  significant  problems  during  their  OT,  analysis  of  the  number  of  problems 
found  by  all  the  programs,  broken  down  by  the  types  of  problems  and  across  the  three  phases  of  OT 

•  Responses  to  Known  Problems  -  For  programs  that  re-observed  known  problems,  analysis  of  the  number  of  programs  that 
identified,  implemented,  and  in  many  cases  tested  fixes  to  these  problems  prior  to  the  OT 

Program-Level 

Analysis  of  the  number 
of  programs  in  which 
problems  have  been 
observed  is  necessary  to 
assess  the  scale  of  problem 
discovery  in  OT.  Note  that 
some  programs,  as  shown 
in  Figure  2,  observed  both 
new  and  known  problems, 
so  these  can  contribute 
to  program  counts  for 
both  types  of  problems. 

The  types  of  problems 
observed  can  be  related  to 
effectiveness,  suitability, 
or  cybersecurity;  programs 
can  observe  multiple  types 
of  problems  during  their 
OT. 

Of  the  15  programs  that 

successfully  completed  OT  in  FY14,  4  programs  delayed  their  OT  in  order  to  fix  performance  problems  prior  to  the  OT. 

Table  3  shows  these  broken  down  by  program  discovery  category,  type  of  problem  observed,  and  phase  of  testing.  A  list  of 
the  programs  that  fall  under  these  categories  in  Table  3  is  included  at  the  end  of  this  section. 

More  than  one-third  of  the  programs  (18  of  48)  undergoing  OT  in  FY14  discovered  new  problems.  When  aggregated,  these 
programs  were  divided  nearly  evenly  between  observing  effectiveness  (11)  and  suitability  (9)  problems,  along  with  a  few 
cybersecurity  (5)  problems.  When  broken  down  by  the  phase  of  testing,  the  ratio  of  programs  discovering  new  suitability 
problems  to  the  discovery  of  new  effectiveness  problems  is  lowest  in  pre-IOT&E  testing.  This  suggests  that  suitability  issues 
manifest  themselves  later  in  the  testing  lifecycle,  but  the  sample  size  is  too  small  to  definitively  state  whether  this  is  a  trend, 
or  simply  random  chance. 

The  proportion  of  programs  re-observing  known  problems  in  OT  remains  high.5  Table  3  shows  that  25  of  the  48  programs 
that  underwent  OT  in  FY14  encountered  known,  significant  problems.  Fifteen  of  the  25  programs  (see  Figure  2)  encountered 
only  known  problems  during  their  OT,  while  the  other  10  also  discovered  new  problems. 

Problem-Level 

Analysis  based  on  the  number  of  problems  observed  during  OT  can  help  characterize  the  completeness  of  testing  prior  to 
OT.  Such  results  are  shown  in  Table  4.  Table  4  is  similar  to  Table  3,  except  it  indicates  the  number  of  problems  encountered 
during  OT  instead  of  the  number  of  programs  encountering  problems.6  Some  of  the  new  problems  observed  were  not 
discoverable  prior  to  commencing  OT  while  others  were.  For  new  problem  discovery,  about  half  of  the  effectiveness 
problems  found  (8  of  1 7)  were  not  discoverable  prior  to  the  OT.  Such  problems  generally  require  the  operationally  realistic 
(or  “test-as-you-fight”)  environment  that  is  the  hallmark  of  OT  in  order  to  be  discovered.  For  new  suitability  problems 
discovered,  this  drops  to  one-third  (3  of  9). 

5.  This  result  is  shown  in  Figure  2,  but  is  not  directly  observable  in  Table  3  because  some  programs  observed  both  new  problems  and  known 
problems. 

6.  The  problems  referred  to  here  are  the  number  of  “significant”  problems,  not  all  problems.  Recall  that  significant  problems  are  those  that  would 
negatively  affect  my  assessment  of  operational  effectiveness,  suitability,  or  cybersecurity. 
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TABLE  3.  FY14  OT  RESULTS  BASED  ON  NUMBER  OF  PROGRAMS 

Category 

Number  of 
Programs1 

Number  of  Programs  by 
Type  of  Problem2 

Number  of  Programs  by  Phase  of  Testing 

Pre-IOT&E 

IOT&E 

FOT&E 

Successful 

OT 

Successful  OT 
without  delays 

11 

1 

6 

4 

Successful,  but 
delayed,  OT 

4 

1 

2 

1 

New  problem  discovery 

18 

Effectiveness 

11 

4 

4 

4 

Suitability 

9 

2 

4 

2 

Cybersecurity 

5 

0 

4 

1 

Total 

5 

8 

5 

Known  problem 
re-observations 

25 

Effectiveness 

13 

4 

4 

5 

Suitability 

14 

3 

7 

4 

Cybersecurity 

4 

1 

1 

2 

Total 

5 

10 

10 

1 .  Forty-eight  programs  underwent  an  OT  in  FY1 4.  Fifteen  had  successful  OTs  (1 1  +  4),  and  33  uncovered  problems.  The  number  of 
programs  that  experienced  new  problem  discovery  or  re-observed  known  problems  adds  up  to  more  than  33  in  the  table  because 
some  programs  experienced  both  new  problem  discovery  and  known  problem  re-observations,  thus  contributing  to  both  counts. 

2.  The  count  of  programs  that  discovered/observed  problems  during  testing  exceeds  the  totals  in  the  "Numbers  of  programs" column 
because  some  programs  discovered  multiple  types  of  problems. 

F  Y  1  4  DOT&E  ACTIVITY  AND  OVERSIGHT 


The  proportion  of  the  number  of 
known  problems  encountered  in  OT 
to  new  problem  discovery  may  be  an 
area  where  improvement  is  possible. 

Table  4  shows  that  42  known  problems 
were  re-observed  in  OT,  compared  to 
27  new  problems  being  discovered 
(of  which  1 1  were  not  discoverable  in 
pre-OT  testing). 

Responses  to  Known  Problems 

As  shown  in  Table  5,  about  half  of  the 
programs  that  commenced  IOT&E  or 
FOT&E  in  FY14  with  one  or  more 
known  effectiveness  problems  did  not 
identify  fixes  to 
address  these 
problems  prior 
to  OT  (both  3  of 
5  in  IOT&E 
and  2  of  5  in 
FOT&E).7  The 
situation  is  better, 
however,  when 
the  program 
commenced 
OT  with  known 
suitability 

problems.  In  this  case,  5  of  7  programs  in  IOT&E,  and  3  of  4  in  FOT&E  implemented  (and  in  many  cases  tested)  fixes 
to  these  problems  prior  to  OT.  Note,  however,  that  by  breaking  down  the  results  thus  far,  the  number  of  programs  in 
each  category  (fix  not  identified  or  fix  implemented  (and  in  many  cases  tested))  is  small.  The  sample  size  is  too  small  to 

definitively  state 
whether  this  is  a  trend 
or  simply  random. 

Specific  Programs  that 
hadOTin  FY14 

Successful  OT 

Fifteen  of  the  48 
programs  that 
underwent  OT  in 
FY14  experienced 
successful  performance 
in  OT.  The  majority 
of  these  (11  of  15) 
that  successfully 
completed  an  OT  event 
did  so  without  having 
to  delay  OT.  These 
programs  are  listed 
below  in  Table  6  and 

are  examples  of  a  successful  development  process,  including  DT  and  OT.  Additional  details  on  any  of  these  programs  can  be 
found  in  the  program-specific  entries  in  the  main  body  of  this  report. 


TABLE  6.  PROGRAMS  THAT  HAD  SUCCESSFUL  OT  IN  FY14 

Successful  OT  (No  Delays) 

Successful  OT  (with  Delays) 

AH-64E 

Cobra  King  (formerly  Cobra  Judy  Replacement) 

AIM-120  Advanced  Medium-Range  Air-to-Air  Missile 
(AMRAAM)1 

Excalibur  Increment  1 B  M982E1 

F/A-18E/F  Super  Hornet  and  EA-18G  Growler 

Global  Command  and  Control  System  -  Joint  (GCCS-J)2 

Global  Command  and  Control  System  -  Maritime  (GCCS-M) 

MQ-9  Reaper  Armed  Unmanned  Aircraft  System  (UAS) 

Integrated  Personnel  and  Pay  System  -  Army  (IPPS-A) 

Joint  Tactical  Network  (JTN) 

MH-60R  Multi-Mission  Helicopter 

MH-60S  Multi-Mission  Combat  Support  Helicopter 

Nett  Warrior 

RQ-7BV2  Shadow  Tactical  Unmanned  Aircraft  System 
(TUAS) 

Department  of  Defense  (DOD)  Teleport 

1 .  This  was  the  Electronic  Protection  Improvement  Program  (EPIP),  a  software  upgrade  to  a  previously-fielded  missile. 

2.  Emerging  results  from  the  OT  have  not  been  completely  analyzed. 

TABLE  5.  ACTIONS  TAKEN  TO  MITIGATE  KNOWN  PROBLEMS  PRIOR  TO  ENTERING  OT 


Category 

Number  of 
Programs 

Number  of  Programs 
by  Type  of  Problem 

How  was  the  problem  addressed  prior  to  OT? 

IOT&E 

FOT&E 

Fix  Not 
Identified1 

Fix 

Implemented 

(tested)2 

Fix  Not 
Identified1 

Fix 

Implemented 

(tested)2 

Known  problem 
re-observations 
(IOT&E  or  FOT&E) 

20 

Effectiveness 

9 

3 

2(1) 

1 

3(1) 

Suitability 

11 

2 

5(3) 

1 

3(3) 

Cybersecurity 

3 

1 

1  (D 

1 

1  (0) 

1 .  Number  of  programs  that  had  at  least  one  problem  for  which  no  fix  was  identified. 

2.  Number  of  programs  that  had  at  least  one  problem  for  which  a  fix  was  implemented  (tested). 


TABLE  4.  FY14  OT  RESULTS  BASED  ON  NUMBER  OF  PROBLEMS 

Category 

Number  of  Problems  by 
Type1 

Number  of  Problems  by  Phase  of  Testing1 

Pre-IOT&E 

IOT&E 

FOT&E 

New  problem  discovery 

Effectiveness 

18(8) 

8(4) 

5(2) 

5(2) 

Suitability 

9(3) 

2(1) 

5(1) 

2(1) 

Total 

27(11) 

10(5) 

10(3) 

7(3) 

Known  problem 
re-observations 

Effectiveness 

19 

7 

7 

5 

Suitability 

23 

5 

11 

7 

Total 

42 

12 

18 

12 

1 .  Numbers  in  parentheses  are  the  number  of  problems  that  were  not  discoverable  prior  to  OT.  For  example,  in  IOT&E, 
five  new  effectiveness  problems  were  identified  in  FY1 4  across  all  programs  undergoing  IOT&E.  Of  these,  two  were 
not  discoverable  prior  to  IOT&E. 

7.  Programs  that  had  known  problems  prior  to  commencing  pre-IOT&E  testing  are  not  counted  here  because  in  most  cases  there  may  be  sufficient 
time  prior  to  starting  IOT&E  to  address  these  problems.  Furthemore,  not  all  required  system  capabilities  might  be  present  for  pre-IOT&E 
events. 
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New  problem  discovery 

Of  the  48  programs  that  underwent  OT  in  FY 14,  18  discovered  problems  that  had  not  been  seen  before.  There  are  a  variety 
of  reasons  why  some  problems  are  not  observed  prior  to  OT.  In  some  cases,  problems  can  be  uncovered  only  by  testing 
under  the  operationally  realistic  conditions  that  characterize  formal  OT.  The  sooner  these  problems  are  discovered,  the  better. 
Hence,  finding  these  problems  in  the  pre-IOT&E  phase  of  OT,  such  as  an  operational  assessment  prior  to  the  Milestone  C 
or  Low-Rate  Initial  Production  decision,  is  highly  desirable.  Other  problems  are  discovered  for  the  first  time  that  could 
have  been  found  and  addressed  during  testing  prior  to  OT,  such  as  dedicated  Developmental  Test  and  Evaluation  (DT&E). 
Table  7  gives  a  list  of  the  programs  that  had  new  problem  discovery  and  indicates  whether  these  problems  were  discoverable 
earlier.  Note  that  many  programs  experience  both  new  problem  discovery  in  OT  and  re-observation  of  known  problems. 
These  programs  are  highlighted  in  grey  in  Table  7.  Additional  details  on  any  of  these  programs  can  be  found  in  the  program- 
specific  entries  in  the  main  body  of  this  report. 


TABLE  7.  PROGRAMS  THAT  HAD  NEW  PROBLEM  DISCOVERY  IN  OT  IN  FY14 

Program 

OT  Event 
Type 

Service 

Discoverable  Prior  to  OT 

Not  Discoverable  Prior  to  OT 

Effectiveness 

Suitability 

Cybersecurity 

Effectiveness 

Suitability 

Cybersecurity 

AN/PRC-1 1 7G 

Pre-IOT&E 

Army 

X 

DOD  Automated  Biometric 
Identification  System  (ABIS) 

Pre-IOT&E 

Army 

X 

X 

Defense  Medical  Information 
Exchange  (DMIX) 

Pre-IOT&E 

Joint 

X 

X 

Joint  Light  Tactical  Vehicle  (JLTV) 
Family  of  Vehicles  (FoV) 

Pre-IOT&E 

Joint 

X 

Surface  ShipTorpedo  Defense 
(SSTD)  System:  Torpedo 

Warning  System  (TWS)  and 
Countermeasure  Anti-Torpedo 
(CAT) 

Pre-IOT&E 

Navy 

X 

Ballistic  Missile  Defense  System 
(BMDS) 

IOT&E 

MDA 

X 

Distributed  Common  Ground 
System  -  Marine  Corps 
(DCGS-MC) 

IOT&E 

Marine 

Corps 

X 

X 

F-1 5E  Radar  Modernization 

Program  (RMP) 

IOT&E 

Air  Force 

X 

Joint  Battle  Command  -  Platform 
(JBC-P) 

IOT&E 

Joint 

X 

Joint  High  Speed  Vessel  (JHSV) 

IOT&E 

Navy 

X 

X 

X 

Miniature  Air-Launched  Decoy 
(MALD)  and  MALD  -  Jammer 
(MALD-J) 

IOT&E 

Air  Force 

X 

Q-53  Counterfire  Target 

Acquisition  Radar  System 

IOT&E 

Army 

X 

X 

RQ-21 A  Blackjack  (formerly  Small 
Tactical  Unmanned  Aerial  System 
(STUAS)) 

IOT&E 

Navy 

X 

AIM-1 20D  Advanced 
Medium-Range  Air-to-Air  Missile 
(AMRAAM) 

FOT&E 

Air  Force 

X 

X 

Air  Operations  Center  -  Weapon 
System  (AOC-WS) 

FOT&E 

Air  Force 

X 

X 

Mark  XIIA  Mode  5  Identification 
Friend  or  Foe  (IFF) 

FOT&E 

Navy 

X 

MK54  Lightweight  Torpedo 

FOT&E 

Navy 

X 

Manpack  Radio 

FOT&E 

Army 

X 

X 
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The  following  discussion  involves  the  discovery  of  new  problems  in  two  of  the  programs  listed  in  Table  7.  The  programs 
are:  (1)  the  DOD  Automated  Biometric  Identification  System  (ABIS)  and  (2)  the  Ballistic  Missile  Defense  System  (BMDS). 
These  two  programs  illustrate  the  value  of  OT  regarding  new  problem  discovery.  Specifically,  the  DOD  ABIS  provides 
a  powerful  example  of  the  benefits  of  testing  in  an  operationally  realistic  environment.  The  BMDS  is  an  example  of  an 
exceedingly  complex  weapon  system  that  discovers  problems  during  operational  flight  testing  (costing  hundreds  of  millions 
of  dollars),  that  should  have  been  found  in  a  more  cost-effective  fashion  through  comprehensive  ground  testing. 

DOD  Automated  Biometric  Identification  System 
(ABIS) 

DOD  ABIS  is  the  result  of  a  Joint  Urgent  Operational 
Need  request  and  consists  of  information  technology 
components  and  biometric  examiner  experts  that 
receive,  process,  and  store  biometrics  from  collection 
assets  across  the  globe,  match  new  biometrics  against 
previously  stored  assets,  and  update  stored  records 
with  new  biometrics  and  contextual  data  to  positively 
identify  and  verify  actual  or  potential  adversaries. 

ABIS  has  been  fielded  and  supported  as  an  Army 
Quick  Reaction  Capability  since  2009.  Since  it  was 
not  a  formal  program  of  record,  ABIS  has  not  had  an 
approved  Test  and  Evaluation  Master  Plan  to  guide 
the  developmental  or  OT&E  of  this  system.  After 
an  initial  deployment  attempt  in  August  2013,  which 
was  unsuccessful,  the  biometrics  program  undertook 
a  set  of  user  tests  that,  while  not  fully  conducted  to  the  rigor  of  a  formal  DT,  were  more  rigorous  than  previous  regression 
testing. 

In  August  2014,  the  Army  Test  and  Evaluation  Command  performed  a  two-phased  OT  on  ABIS  version  1.2.  This  was 
the  first  OT  conducted  on  the  system.  The  first  phase  was  conducted  August  7  -  28,  2014.  The  second  phase,  which  was 
supposed  to  begin  directly  following  the  first  phase,  was  delayed  to  address  the  five  effectiveness  problems  discussed 
below.  The  second  phase  of  OT  was  conducted  October  17  -  22,  2014. 

The  first  phase  of  OT  was  structured  to  allow  comparison  between  the  then  current  Authoritative  Database  (ABIS  1.0)  and 
the  system  under  test  (ABIS  1.2)  by  streaming  all  live  data  into  both  systems.  To  mitigate  operational  risk,  only  ABIS  1.0 
sent  responses  back  to  the  field.  Since  the  ABIS  1 .2  system  was  not  the  authoritative  system,  Phase  1  of  the  OT  could  have 
been  conducted  as  an  operationally  realistic  DT  event. 

During  the  first  phase  of  OT,  the  following  problems  were  noted  in  a  DOT&E  memo  to  Army  acquisition  leadership.  If  a 
rigorous  DT  using  an  operationally  realistic  environment  had  been  conducted  prior  to  the  OT,  the  problems  detailed  in  the 
DOT&E  memo  after  Phase  1  of  the  OT  would  have  likely  been  identified.  The  issues  are  detailed  as  follows: 

•  The  National  Ground  Intelligence  Center  (NGIC)  puts  all  ABIS  biometric  match  results  into  its  Biometric  Identity 
Intelligence  Repository  (BI2R).  NGIC  also  fuses  the  data  from  worldwide  biometric  collection  systems  into  and  out  of 
ABIS.  BI2R  is  used  by  DOD  Intelligence  agencies  to  identify  persons  that  should  be  added  to  the  watchlist.  During 
Phase  1  OT,  NGIC  observed  thousands  of  discrepancies  between  match  results  returned  from  ABIS  1.0  and  1.2.  Other 
problems  included  incorrect  email  addresses  for  sending  alerts.  Without  alerts,  no  actions  can  be  taken  when  a  person 
on  a  watchlist  is  identified  by  the  system.  The  mission  impact  is  the  potential  loss  of  actionable  intelligence  when 
encountering  persons  of  interest  throughout  the  world.  OT  was  necessary  to  uncover  this  problem  because  the  number 
and  complexity  of  live  interfaces  with  real-world  biometric  submitters  could  not  be  adequately  simulated  in  a  DT. 

•  A  latent  fingerprint  is  one  taken  from  an  object  in  the  field,  such  as  an  improvised  explosive  device.  Latent  (fingerprint) 
examiners  at  the  Biometrics  Identity  Management  Activity  noted  a  key  identifier  (Grand  ID)  was  missing  from  Latent 
Examination  tools  in  ABIS  1.2.  This  capability  was  available  in  ABIS  1.0.  The  Grand  ID  enables  latent  examiners  to 
link  different  latent  images  with  a  single  forensic  case.  The  problem  was  not  discoverable  prior  to  entering  OT  because 
the  user  cases  that  were  exercised  required  external  interfaces  with  biometric  and  latent  submitters  that  could  not  be 
simulated  in  the  DT  environment. 

•  ABIS  1.2  responses  to  biometric  submissions  failed  to  meet  the  specifications  required  by  the  Federal  Bureau  of 
Investigation  Integrated  Automated  Fingerprint  Identification  System  (IAFIS)  preventing  acceptance  by  the  IAFIS 
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interface.  An  operational  environment  with  the  actual  production  equipment  receiving  submissions  in  parallel  with  the 
legacy  operational  system  was  essential  to  allow  discovery  of  such  issues. 

•  One  of  four  custom  watchlists  had  over  1,800  discrepancies  between  the  responses  from  ABIS  1.0  and  1.2.  Custom 
watchlists  are  used  by  military  personnel  in  the  field  to  determine  courses  of  action  when  a  person  is  detained  in  a 
particular  geographic  area.  Custom  Biometrically  Enabled  Watchlists  could  have  been  assessed  before  the  Phase  1  OT 
began  while  the  live  submissions  were  streaming  into  both  systems. 


Ballistic  Missile  Defense  System 
(BMDS) 

The  BMDS  is  designed  to  protect  the 
United  States,  deployed  forces,  allies, 
and  friends  against  ballistic  missiles  of 
all  ranges  and  in  all  phases  of  flight.  The 
BMDS  is  a  distributed  system  currently 
comprised  of  five  elements  (four  shooters 
and  one  command  and  control  element) 
and  five  sensor  systems  (four  radar 
systems  and  one  space-based  system). 

The  first  OT  of  the  BMDS,  referred 
to  as  Flight  Test  Operational  -  01 
(FTO-01)  occurred  in  September  2013 
and  demonstrated  a  layered  upper-tier 
regional/theater  BMDS  defense  against 
a  raid  of  two  simultaneously-launched  and  threat-representative  medium-range  ballistic  missiles  threatening  a  shared 
defended  area.  Although  a  layered  defense  was  demonstrated  in  this  test,  true  system  integration  was  not  demonstrated  due 
to  system  network  configuration  errors,  interoperability  limitations,  and  component  failures. 


FTO-01  was  an  extremely  complex  flight  test  event — it  was  the  second  most  complex  flight  test  ever  attempted  by  the 
Missile  Defense  Agency  (MDA)  to  date.  A  major  difficulty  in  finding  problems  such  as  those  uncovered  during  FTO-01 
is  that  the  BMDS  can  be  instantiated  in  many  ways  using  different  combinations  of  shooters,  sensors,  and  operational 
laydowns.  Despite  this  variability,  some  of  these  findings  could  have  been  discovered  prior  to  executing  the  flight  test.  In 
particular,  some  of  the  network  configuration  errors  could  have  been  discovered  through  comprehensive  ground  testing 
and  analyses.  The  MDA  has  taken  action  to  correct  the  problems  uncovered  during  FTO-01.  Details  of  the  problems  and 
specific  actions  are  classified. 
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Known  problem  re-observations 

Some  problems  are  observed  in  OT  that  are  already  known  from  prior  testing;  known  problems  were  observed  in  25  of  the 
48  programs  that  underwent  OT  in  FY14.  As  noted  earlier,  many  programs  that  re-observed  known  problems  also  experienced  new 
problem  discovery  in  OT;  these  are  highlighted  in  grey  in  Table  8. 


TABLE  8.  PROGRAMS  WITH  KNOWN  PROBLEM  RE-OBSERVATIONS 

Program 

OT  Event 
Type 

Service 

Known  Problem 

Effectiveness 

Suitability 

Cybersecurity 

AN/PRC-1 17G 

Pre-IOT&E 

Army 

X 

AN/SQQ-89A(V)15  Integrated  Undersea  Warfare  (USW) 
Combat  System  Suite 

Pre-IOT&E 

Navy 

X 

CVN-78  Gerald  R.  Ford  class  Nuclear  Aircraft  Carrier 

Pre-IOT&E 

Navy 

X 

X 

Defense  Enterprise  Accounting  and  Management  System 
(DEAMS) 

Pre-IOT&E 

Air  Force 

X 

X 

X 

Infrared  Search  and  Track  (IRST) 

Pre-IOT&E 

Navy 

X 

Aegis  Ballistic  Missile  Defense  (BMD) 

IOT&E 

Navy, 

MDA 

X 

F-1 5E  Radar  Modernization  Program  (RMP) 

IOT&E 

Air  Force 

X 

Joint  Battle  Command  -  Platform  (JBC-P) 

IOT&E 

Joint 

X 

Littoral  Combat  Ship  (LCS)1 

IOT&E 

Navy 

X 

X 

X 

Miniature  Air  Launched  Decoy  (MALD)  and  MALD -Jammer 
(MALD-J) 

IOT&E 

Air  Force 

X 

Multi-Static  Active  Coherent  (MAC)  System 

IOT&E 

Navy 

X 

Q-53  CounterfireTarget  Acquisition  Radar  System 

IOT&E 

Army 

X 

QF-16  Full-Scale  AerialTarget  (FSAT) 

IOT&E 

Air  Force 

X 

RQ-21 A  Blackjack  (formerly  Small  Tactical  Unmanned  Aerial 
System  (STUAS)) 

IOT&E 

Navy 

X 

Surveillance  Towed  Array  Sensor  System  (SURTASS)  and 
Compact  Low  Frequency  Active  (CLFA)  Sonar 

IOT&E 

Navy 

X 

Air  Force  Distributed  Common  Ground  System  (AF  DCGS) 

FOT&E 

Air  Force 

X 

Air  Operations  Center  -  Weapon  System  (AOC-WS) 

FOT&E 

Air  Force 

X 

Battle  Control  System  -  Fixed  (BCS-F) 

FOT&E 

Air  Force 

X 

F/A-1 8E/F  Super  Hornet  and  EA-1 8G  Growler 

FOT&E 

Navy 

X 

X 

Joint  Warning  and  Reporting  Network  (JWARN) 

FOT&E 

Joint 

X 

Mark  XIIA  Identification  Friend  or  Foe  (IFF)  Mode  5 

FOT&E 

Navy 

X 

MK  54  Lightweight  Torpedo 

FOT&E 

Navy 

X 

MV-22  Osprey 

FOT&E 

Joint 

X 

P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA) 

FOT&E 

Navy 

X 

Manpack  Radio 

FOT&E 

Army 

X 

X 

1  .  Two  survey  entries  for  separate  oversight  programs  with  separate  problems,  both  discussed  in  the  LCS  section  of  the  annual  report. 
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The  following  discussion  involves  the  re -observation  of  known  problems  in  three  of  the  programs  listed  in  Table  8.  The 
programs  are:  (1)  Air  Force  Distributed  Common  Ground  System  (AF  DCGS);  (2)  Multi-static  Active  Coherent  (MAC) 
System;  and  (3)  Small  Tactical  Unmanned  Aerial  System  (STUAS)  Tier  II.  These  three  programs  illustrate  the  value  of  OT 
in  highlighting  the  operational  implications  of  known  problems.  In  these  three  cases,  program  management  either  decided  to 
accept  the  risk  that  their  known  problems  would  not  affect  the  OT  assessment,  or  let  schedule  drive  the  program  into  OT  in 
spite  of  known  shortcomings. 


/  JSTARS 


Coalition  Assets 


Other  Service  DCGS^, 


H 


Air  Force  Distributed  Common  Ground  System  (AF  DCGS) 

The  AF  DCGS  provides  software  tools  for  operators 
to  task,  process,  exploit,  and  disseminate  Intelligence, 

Surveillance,  and  Reconnaissance  information. 

AF  DCGS  consists  of  multiple  ground  systems  at 
dispersed  operational  sites.  AF  DCGS  participates 
in  the  DOD  intelligence  enterprise  via  the  DCGS 
Integration  Backbone,  which  uses  a  metadata  catalog 
and  discovery  service  to  enable  sharing  of  information 
among  participants. 

AF  DCGS  Bulk  Release  10B  failed  both  developmental 
and  regression  testing  and  did  not  meet  the  entrance 
criteria  for  the  OT  phase  known  as  the  Force 
Development  Evaluation.  Despite  not  meeting  the  OT 
entrance  criteria  (the  system  had  two  known  Category 
I  and  four  Category  II  software  deficiencies  that  were 
open  and  unresolved),  the  Air  Force  Intelligence, 

Surveillance,  and  Reconnaissance  Agency  approved 
entrance  into  OT.  In  January  and  June  2014,  the  605th 
Test  and  Evaluation  Squadron  conducted  Phases  1 
and  2  of  a  two-phase  Force  Development  Evaluation 
to  assess  the  operational  effectiveness  and  suitability 
of  AF  DCGS  Bulk  Release  10B.  Two  new  software  applications  that  were  part  of  the  Geospatial  Intelligence  upgrade 
known  as  Bulk  Release  10B  had  major  performance  problems.  They  caused  such  significant  slowdowns  in  workflow  that 
the  Air  Force  made  the  decision  to  stop  using  the  new  applications,  and  operators  reverted  to  using  the  legacy  manual 
processes  during  the  test.  The  system  did  not  meet  any  of  its  reliability  requirements  because  of  critical  failures  and 
downtime.  While  users  can  execute  their  missions  with  AF  DCGS  under  normal  load  conditions,  performance  under  heavy 
loads  could  not  be  determined.  Heavier  loads  are  expected  in  the  future  when  new  sensors  are  deployed  and  the  number  of 
simultaneous  external  users  is  increased. 


AOC  -  Air  Operations  Center  JIOC  •  Joint  Intelligence  Operations  Center 
AWACS  •  Airborne  Warning  and  JSTARS  -  Joint  Surveillance  Target  Attack 
Control  System  Radar  System 

GH  -  Global  Hawk  JTF  -  Joint  Task  Force 

HQ  -  Headquarters  TARS  -  Tactical/Theater  Airborne 

JFC  -  Joint  Forces  Command  Reconnaissance  System 


In  part  because  the  Air  Force  placed  AF  DCGS  in  the  sustainment  vice  development  phase,  the  program  lacks  a  strategy  for 
testing  and  evaluation,  documented  performance  requirements  for  planned  enhancements,  accurate  software  maturity  trend 
information,  and  an  approved  system-engineering  plan.  By  developing  and  following  these  key  programmatic  guidance 
documents,  the  Air  Force  would  likely  improve  AF  DCGS  performance. 

Multi-static  Active  Coherent  (MAC)  System 

The  MAC  system  is  an  active  sonar  system  composed 
of  two  types  of  sonobuoys  (source  and  receiver)  and 
an  acoustic  processing-software  suite.  It  is  employed 
by  the  Navy’s  maritime  patrol  aircraft  (P-3Cs  and 
eventually  P-8As)  to  search  for  and  locate  threat 
submarines  in  a  variety  of  ocean  conditions. 

The  Navy  completed  OT  of  the  MAC  Phase  1 
system  on  P-3C  Multi-mission  Aircraft  (MMA)  in 
October  2013.  OT  consisted  of  3  DT  events  conducted 
off  the  coast  of  Jacksonville,  Florida;  7  dedicated  OT 
events  conducted  in  the  Southern  California  Fleet 
Operating  Areas;  and  14  events  in  the  Narragansett  Bay 
Operating  Areas.  After  the  series  of  OT  in  January  2013  near  San  Diego,  the  Navy  knew  the  system  did  not  work  in  some 
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environments.  However,  the  program  requirement  criterion  for  MAC  was  a  roll-up  of  test  detection  results,  and  testing  was 
not  stopped  until  a  series  of  test  events  in  May  2013  in  the  Narragansett  Bay  test  area,  where  performance  appeared  below 
threshold,  even  after  counting  DT  results  prior  to  2013. 

During  the  May  2013  series  of  tests,  it  became  clear  operators  were  not  recognizing  valid  target  returns  as  targets  because 
the  target-signature  criteria  they  had  been  trained  to  use  did  not  cover  the  new  environment.  There  was  also  a  need  to  fix 
some  materiel  information  technology  problems  with  the  aircraft  used  for  the  test.  OT  was  halted  for  a  period  of  four  to 
five  months  because  of  a  combination  of  materiel  problems  with  the  aircraft  used  to  employ  the  system  and  training  of 
operators  to  use  the  system.  More  thorough  DT  might  have  minimized  or  eliminated  this  delay.  The  operators  were 
retrained  to  recognize  new  target  signature  features  that  enabled  them  to  distinguish  between  valid  target  returns  and  clutter 
returns  more  effectively.  In  the  October  2013  test  series  following  the  re -training,  the  operators  were  able  to  recognize 
valid  targets  more  accurately,  but  not  by  a  margin  that  coidd  be  clearly  distinguished  from  previous  rates  under  the 
confidence  limits  of  the  data  collected. 

RQ-21A  Blackjack  (formerly  Small  Tactical  Unmanned  Aerial  System  (STUAS)  Tier  II ) 

Marine  Corps  commanders  will  use  the  RQ-21 A  Blackjack  (formerly  Small  Tactical  Unmanned  Aerial  System  (STUAS)) 
to  provide  units  ashore  with  a  dedicated  persistent  battlefield  Intelligence,  Surveillance,  and  Reconnaissance  (ISR) 
capability  that  will  reduce  their  dependence  on  higher  headquarters  for  ISR  support.  The  persistence  of  the  system  allows 
commanders  greater  coverage  of  their  areas  of  interest, 
while  providing  the  capability  to  concentrate  for  longer 
periods  of  time  on  a  specified  target  of  interest.  The 
Marine  Corps  is  developing  RQ-21  A  as  an  organic  asset 
in  an  effort  to  wean  itself  off  the  contractor-owned, 
contractor-operated  systems  currently  under  contract, 
in  order  to  transition  from  ISR  services  contracts  to  an 
organic  ISR  asset,  the  Program  Office  decided  to  enter 
IOT&E  in  spite  of  the  low  reliability  demonstrated 
during  an  earlier  operational  assessment. 

The  Navy  started  the  RQ-21  A  IOT&E  in  January  2014. 

Testing  consisted  of  a  land-based  IOT&E  phase  (with 
concurrent  ship-based  DT)  intended  to  be  followed  by 
a  ship-based  IOT&E  phase  aboard  an  LPD-17  class 
ship.  During  the  land-based  phase  of  IOT&E  at  Marine 
Corps  Air  Ground  Combat  Center,  Twentynine  Palms,  California,  operators  flew  188  flight  hours  during  31  flights.  The 
first  flight  ended  in  a  mishap  and  loss  of  the  air  vehicle.  Post-mishap  investigation  suspended  OT  flights  for  10  days.  The 
RQ-21A  demonstrated  a  Mean  Flight  Hours  Between  Abort  (MFHBA)  of  15.8  hours,  well  below  the  MFHBA  threshold 
criterion  of  50  hours.  Low  reliability  adversely  affected  the  ability  of  operators  to  support  ground  units  in  a  timely  manner. 
Many  of  the  reliability  problems  identified  during  the  land-based  IOT&E  appear  to  result  from  poor  quality  control  during 
the  production  process.  The  Program  Office  is  working  with  the  manufacturer  to  increase  quality  control  processes  with 
sub-vendors,  improve  acceptance  testing  of  spare  parts,  and  review  their  acceptance  procedures. 

Concurrent  with  the  land-based  phase  of  IOT&E,  the  Navy  conducted  RQ-21A  ship-based  DT  aboard  an  LPD-17  class 
ship.  This  ship  testing  identified  interference  between  the  ship’s  degaussing  system  and  the  air  vehicle’s  magnetometer. 
Without  realistic  shipboard  testing,  this  deficiency  would  not  have  been  identified.  This  deficiency  necessitated  software 
upgrades  and  regression  testing,  which  delayed  the  scheduled  ship-based  phase  of  IOT&E  until  December  2014.  Based 
on  poor  system  performance  during  the  land-based  phase  of  IOT&E  and  software  update  to  correct  a  GPS  deficiency 
associated  with  shipboard  operations,  the  Navy  conducted  a  second  land-based  phase  of  IOT&E  in  June  at  Marine  Corps 
Base  Camp  Lejeune,  North  Carolina.  Operators  flew  20.9  hours  during  eight  flights.  Analysis  of  results  is  ongoing. 

For  programs  with  upcoming  OT  events  in  the  next  three  years,  I  found  that  slightly  more  than  one-third  (15  of  42)  of  the 
programs  currently  do  not  exhibit  performance  problems  significant  enough  to  jeopardize  successful  performance  in  OT. 
Table  9  shows  these  results  by  type  of  problem  and  phase  of  testing. 
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PROGRAMS  WITH  UPCOMING  OT  EVENTS 

For  programs  with  upcoming 
OT  events  in  the  next  three 
years,  I  found  that  slightly 
more  than  one-third  (15  of 
42)  of  the  programs  currently 
do  not  exhibit  performance 
problems  significant  enough 
to  jeopardize  successful 
performance  in  OT.  Table  9 
shows  these  results  by  type  of 
problem  and  phase  of  testing. 

Upcoming  pre-IOT&E  test 
events  are  far  more  likely  to 
be  delayed  to  correct  problems 
compared  to  both  upcoming 
IOT&E  and  FOT&E.  In 
fact.  Table  9  shows  that  for 
the  programs  covered  in  this 
Annual  Report,  there  were 
no  upcoming  FOT&E  events 
that  were  delayed  to  correct 

problems.  Of  the  programs  for  which  potential  problems  exist  for  upcoming  OT  events,  5  of  7  of  the  pre-IOT&E  events  were 
delayed  to  address  at  least  one  issue,  2  of  10  of  the  IOT&E  events  were  delayed,  and  0  of  6  of  the  FOT&E  (or  post-IOT&E) 
events  were  delayed. 

For  programs  that  have  not  delayed  their  upcoming  OT  and  have  known  problems,  the  distribution  between  effectiveness 
and  suitability  problems  is  about  the  same  (13  compared  to  10),  with  three  cybersecurity  problems.  Table  10  expands  further 

upon  programs  that 
have  not  delayed 
their  upcoming 
OT  and  shows  that 
FOT&E  events  are 
considerably  more 
likely  to  have  a  fix 
implemented  (and  in 
many  cases  tested) 
going  into  the  OT, 
regardless  of  type  of 
problem,  compared 
to  both  pre-IOT&E 
and  IOT&E.  The 

data  are  currently  insufficient  to  determine  whether  the  differences  between  the  rate  of  implementing  and  testing  fixes  prior  to 
FOT&E  compared  to  IOT&E  is  a  trend  or  simply  random. 

Specific  Programs  that  have  upcoming  OT  in  the  next  three  years 

No  problems  for  upcoming  OT 

Fifteen  of  the  42  programs  with  upcoming  OT  events  have  not  yet  exhibited  problems  considered  to  significantly  jeopardize 
performance  in  upcoming  OT  events.  Such  programs  could  be  examples  where  the  program  development  process,  including 
DT  and  OT,  appears  to  be  moving  along  well.  Alternatively,  the  testing  to  date  might  not  have  been  sufficiently  stressing  to 
surface  any  problems.  These  programs  are  listed  below  in  Table  1 1 . 


TABLE  1 0.  ACTIONS  TAKEN  TO  ADDRESS  PROBLEMS  FOR  UPCOMING  OT  EVENTS 
FOR  WHICH  OT  HAS  NOT  (YET)  BEEN  DELAYED 


Category 

Type  of  Problem 

How  has  the  problem  been  addressed  to  date  (program  count)? 

pre-IOT&E 

IOT&E 

FOT&E 

Fix  Not 
Identified 

Fix 

Documented 

Fix 

implemented 

(tested) 

Fix  Not 
Identified 

Fix 

Documented 

Fix 

implemented 

(tested) 

Fix  Not 
Identified 

Fix 

Documented 

Fix 

implemented 

(tested) 

Problems  have 
not  delayed 
upcoming  OT' 

Effectiveness 

2 

0 

1  (0) 

4 

4 

2(1) 

2 

0 

2(2) 

Suitability 

0 

1 

0 

4 

1 

1  (0) 

0 

1 

3(2) 

Cybersecurity 

1 

0 

0 

0 

0 

1  (0) 

0 

0 

1  (D 

1 .  Numbers  in  parentheses  are  the  number  of  problems  that  have  already  tested  fixes  prior  to  the  upcoming  OT. 


TABLE  9.  PROGRAMS  COMMENCING  OT  WITHIN  THE  NEXTTHREE  YEARS 


Category 

Number  of 
Programs' 

Number  and  Type  of 
Problems  (program  count)2 

Phase  of  Testing  (program  count) 

Pre-IOT&E 

IOT&E 

FOT&E 

No  problems  for 
upcoming  OT 

15 

4 

5 

6 

Problems  have 
delayed  upcoming  OT 

7 

Effectiveness 

11 

3 

0 

0 

Suitability 

9 

4 

2 

0 

Cybersecurity3 

5 

0 

0 

0 

Total 

5 

2 

0 

Problems  have  not 
delayed  upcoming  OT 

19 

Effectiveness 

2 

7 

4 

Suitability 

1 

5 

4 

Cybersecurity3 

1 

1 

1 

Total 

4 

8 

6 

Other  problems 
threaten  upcoming  OT 

4 

2 

2 

0 

1 .  Forty-two  programs  will  undergo  an  OT  in  the  next  three  years.  The  number  of  programs  adds  up  to  more  than  42  because 
some  programs  have  problems  that  delayed  their  upcoming  OT  as  well  as  problems  that  did  not  delay  OT. 

2.  The  number  of  programs  summed  across  type  of  problems  adds  up  to  more  than  the  number  of  programs  because  some 
programs  have  multiple  problems  or  more  than  one  type  of  problem.. 


24  Problem  Discovery  Affecting  OT&E 


F  Y  1  4  DOT&E  ACTIVITY  AND  OVERSIGHT 


Problems  delayed  upcoming  OT 

For  some  programs,  early  testing  has  uncovered 
problems  and  entry  into  the  upcoming  OT  has 
already  been  delayed  to  provide  the  program  an 
opportunity  to  correct  them.  Seven  programs  fall  into 
this  category  and  are  given  in  Table  12.  Note  that 
some  programs  that  have  problems  that  delayed  their 
upcoming  OT  also  have  problems  that  did  not  delay 
the  OT;  the  programs  with  both  types  of  problems  are 
highlighted  in  grey. 

Problems  have  not  delayed  upcoming  OT 

Some  programs  have  uncovered  problems  in  early 
testing  that,  if  not  satisfactorily  corrected,  could  result 
in  my  assessing  the  system  as  not  being  operationally 
effective  or  suitable.  Unlike  the  above,  the  OT  has 
not  (yet)  been  delayed  to  correct  these  problems. 
These  programs  are  also  shown  in  Table  12.  Note 
that  some  programs  that  have  identified  problems  that 
did  not  delay  their  upcoming  OT  have  also  identified 
other  problems  that  did  delay  the  OT;  the  programs 
with  both  types  of  problems  are  highlighted  in  grey. 


TABLE  1 1 .  PROGRAMS  WITH  NO  PROBLEMS  FOR  UPCOMING  OT 

AIM-9X  -  Air-to-Air  Missile  Upgrade 

Air  and  Missile  Defense  Radar  (AMDR) 

C-1 7  Increase  Gross  Weight  (IGW)  and  Formation  Spacing  Reduction  (FSR) 

Common  Aviation  Command  and  Control  System  (CAC2S) 

E-2D  Advanced  Hawkeye 

F-22A  Advanced  Tactical  Fighter 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T) 

Guided  Multiple  Launch  Rocket  System  -  Alternative  Warhead  (GMLRS-AW)  XM30E1 1 

Integrated  Defensive  Electronic  Countermeasures  (IDECM) 

Joint  Information  Environment  (JIE)2 

Joint  Standoff  Weapon  (JSOW) 

Massive  Ordnance  Penetrator  (MOP) 

Rifleman  Radio 

Small  Diameter  Bomb  (SDB) 

Standard  Missile-6  (SM-6) 

1 .  Emerging  Results  from  the  recent  GMLRS-AW  IOT&E  indicate  its  lethality  is  insufficient. 

2.  The  JIE  has  not  had  any  OT  to  date  pending  development  of  governance  processes  for  the  Joint 
Regional  Security  Stack  transport  infrastructure. 

TABLE  12.  PROGRAMS  WITH  PROBLEMS  THREATENING  UPCOMING  OT 

Program 

OT  Event 
Type 

Service 

OT  Delayed 

OT  Not  Delayed 

Effectiveness 

Suitability 

Cybersecurity 

Effectiveness 

Suitability 

Cybersecurity 

AC-130J  Ghostrider 

Pre-IOT&E 

USSOCOM 

X 

Distributed  Common  Ground  System  -  Army  (DCGS-A) 

Pre-IOT&E 

Army 

X 

X 

F-35  Joint  Strike  Fighter  (JSF) 

Pre-IOT&E 

Joint 

X 

X 

Key  Management  Infrastructure  (KMI) 

Pre-IOT&E 

NSA 

X 

MQ-4C  Triton  Unmanned  Aircraft  System 

Pre-IOT&E 

Navy 

X 

Public  Key  Infrastructure  (PKI) 

Pre-IOT&E 

Joint 

X 

Remote  Minehunting  System  (RMS) 

Pre-IOT&E 

Navy 

X 

X 

X 

AN/SQQ-89A(V)15  Integrated  Undersea  Warfare  (USW)  Combat 
System  Suite 

IOT&E 

Navy 

X 

Ballistic  Missile  Defense  System  (BMDS) 

IOT&E 

MDA 

X 

Defense  Enterprise  Accounting  and  Management  System  (DEAMS) 

IOT&E 

Air  Force 

X 

X 

Infrared  Search  and  Track  (IRST) 

IOT&E 

Navy 

X 

LHA-6  New  Amphibious  Assault  Ship 

IOT&E 

Navy 

X 

Littoral  Combat  Ship  (LCS) 

IOT&E 

Navy 

X 

X 

X 

Mobile  Landing  Platform  (MLP)  Core  Capability  Set  (CCS)  and 
Afloat  Forward  Staging  Base  (AFSB) 

IOT&E 

Navy 

X 

Patriot  Advanced  Capability-3  (PAC-3) 

IOT&E 

Army 

X 

X 

Precision  Guidance  Kit  (PGK) 

IOT&E 

Army 

X 

RQ-4B  Global  Hawk  High-Altitude  Long-Endurance  Unmanned 
Aerial  System  (UAS) 

IOT&E 

Air  Force 

X 

X 

AGM-88E  Advanced  Anti-Radiation  Guided  Missile  (AARGM) 

FOT&E 

Navy 

X 

Air  Force  Distributed  Common  Ground  System  (AF  DCGS) 

FOT&E 

Air  Force 

X 

Defense  Medical  Information  Exchange  (DMIX) 

FOT&E 

Joint 

X 

X 

M829E4  Armor  Piercing,  Fin  Stabilized,  Discarding  Sabot-Tracer 
(APFSDS-T) 

FOT&E 

Army 

X 

MQ-9  Reaper  Armed  Unmanned  Aircraft  System  (UAS) 

FOT&E 

Air  Force 

X 

Warfighter  Information  Network -Tactical  (WIN-T) 

FOT&E 

Army 

X 

X 

X 
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The  following  discusses  the  problems  that  potentially  jeopardize  successful  performance  in  upcoming  OT  in  three  of  the 
programs  listed  in  Table  12.  The  programs  are:  (1)  AC-130J  Ghostrider;  (2)  Remote  Minehunting  System  (RMS);  and 
(3)  Warfighter  Information  Network  -  Tactical  (WIN-T).  These  programs  illustrate  the  types  of  problems  that  jeopardize 
successful  performance  in  upcoming  OT  and  should  be  addressed  to  the  maximum  extent  possible  prior  to  OT. 

AC-130J  Ghostrider 

The  AC-130J  is  a  medium-sized,  multi-engine,  tactical 
aircraft  with  a  variety  of  sensors  and  weapons  for  close  air 
support.  U.S.  Special  Operations  Command  is  developing 
AC-130J  through  the  integration  of  a  modular  Precision 
Strike  Package  (PSP)  onto  existing  MC-130J  aircraft.  The 
PSP  provides  a  30  mm  side-firing  gun;  wing-mounted, 

GPS-guided  Small  Diameter  Bombs;  Griffin  laser-guided 
missiles;  two  electro-optical/infrared  sensor/laser  designator 
pods;  a  synthetic  aperture  radar  pod;  and  multiple  video,  data, 
and  communication  links. 

There  have  been  problems  with  integration  of  the  PSP 
weapon  kit  onto  the  aircraft  that  continue  to  delay  portions  of  DT  by  prohibiting  weapons  employment  and  hindering 
system  effectiveness.  First,  the  visual  acuity  of  the  electro-optical/infrared  sensors  installed  on  the  AC-130J  is  not 
sufficient  for  accurate  target  identification  and  designation  because  of  excessive  vibration  on  the  new  aircraft  as 
compared  to  the  legacy  AC-130W  aircraft  on  which  the  PSP  was  previously  installed.  Second,  electrical/radio-frequency 
interference  between  aircraft  systems  and  the  hand  controllers  used  by  crewmembers  to  direct  the  sensors  and  weapons 
has  caused  erratic  sensor  movements.  This  inhibits  target  tracking  and  is  a  safety  hazard  (risk  of  fratricide)  during  weapon 
employment.  The  program  is  working  on  correcting  the  sensor  vibration  issue  by  collecting  flight  test  data  that  can  be 
used  by  the  subsystem  contractor  to  develop  mechanical  and  software  updates  to  reduce  the  effect  of  vibration.  Similar 
efforts  are  underway  to  characterize  and  correct  electrical  interference  with  the  controllers.  The  program  has  reported 
some  progress  in  the  laboratory  environment  on  both  fixes,  but  definitive  solutions  have  not  yet  been  demonstrated  on  the 
aircraft. 

The  program  has  accomplished  36  test  flights  out  of  approximately  130  flights  planned  for  a  total  of  97  flight  hours.  Initial 
DT  is  now  expected  to  be  completed  in  May  2015.  Delays  in  DT  have  delayed  the  planned  operational  assessment  by  the 
18th  Flight  Test  Squadron  by  approximately  four  months,  and  IOT&E  has  been  delayed  until  October  2015.  This  schedule 
does  not  allow  much  time  to  developing  and  implementing  fixes  to  problems  already  observed  in  the  DT. 

Remote  Minehunting  System  (RMS) 

The  RMS  is  a  system-of-systems  designed  to  detect  and  classify 
mine-like  objects  throughout  the  water  column  and  to  identify 
bottom  objects  in  shallow  waters.  The  Navy  expects  to  employ 
the  system  with  both  variants  of  the  Littoral  Combat  Ship  (LCS) 
as  a  key  component  of  the  Mine  Countermeasures  (MCM)  mission 
package. 

DOT&E  disapproved  the  Navy’s  plan  to  conduct  an  operational 
assessment  of  the  RMS  in  2QFY14  because  the  assessment  would 
have  been  a  wasted  effort  for  the  following  reasons: 

•  The  proposed  test  article  was  not  representative  of  the  system  the 
Navy  plans  to  employ  in  the  first  increment  of  the  LCS  MCM 
mission  package  (it  was  an  earlier  version  without  planned 
upgrades)  and  therefore,  would  not  provide  data  necessary  to 
augment  the  IOT&E  of  an  LCS  equipped  with  that  mission  package; 

•  Test  limitations  would  have  precluded  an  operational  evaluation  of  some  phases  of  the  end-to-end  mission;  and 

•  Conduct  of  the  test  would  have  delayed  vehic  le  upgrades  necessary  to  support  testing  of  the  system  the  Navy  expects  to 
field. 

The  RMS  program  has  not  yet  demonstrated  that  the  system  can  meet  its  detection  and  classification  requirements  against 
moored  and  bottom  mines  spanning  the  portion  of  the  shallow  water  regime  not  covered  by  the  Airborne  Laser  Mine 
Detection  System  (ALMDS).  The  program  anticipates  that  the  AN/AQS-20B  sensor  will  permit  the  system  to  cover  the 
portion  of  the  water  column  below  that  covered  by  the  ALMDS.  The  new  sensor  will  be  tested  in  FY15. 
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RMS  radios  have  had  difficulty  establishing  reliable  communications  with  the  LCS  during  DT,  and  once  communications 
are  established,  the  current  communications  systems  do  not  support  Remote  Multi-Mission  Vehicle  (RMMV)  mine 
identification  operations  beyond  the  horizon.  RMMV  will  need  to  operate  beyond  the  horizon  to  support  efficient  MCM 
operations  in  long  shipping  channels  while  LCS  remains  in  an  area  clear  of  mines.  This  problem  arose  when  the  Navy 
decertified  the  MH-60S  helicopter  for  towing  MCM  devices,  including  the  AN/AQS-20A/B  sensor.  The  range  limitation 
did  not  exist  when  the  sensor  was  towed  by  the  helicopter.  The  Navy  has  not  subsequently  developed  a  solution  to  this 
problem. 

The  combined  results  of  shore -based  and  LCS-based  testing  conducted  since  the  program  was  recertified  following  a 
Nunn-McCurdy  breach  in  2010  have  not  demonstrated  that  an  LCS  equipped  with  an  MCM  mission  package  that  includes 
two  RMMVs  and  three  AN/AQS-20A  sonars  will  be  able  to  support  the  sustained  area  coverage  rate  the  Navy  has 
established  for  the  Increment  1  MCM  mission  package.  The  program  believes  that  RMMV  reliability  improvements  and 
an  upgraded  version  of  the  minehunting  sensor,  designated  AN/AQS-20B,  will  resolve  many  of  the  program’s  identified 
problems. 

The  reliability  of  the  version  4.2  (v4.2)  RMMV  during  combined  developmental  and  integrated  testing  completed  in  FY14 
was  31.3  hours  Mean  Time  Between  Operational  Mission  Failure  (MTBOMF),  which  is  well  below  the  required  reliability. 
DT  completed  in  1QFY15  provides  a  point  estimate  for  v6.0  vehicle  reliability  of  34.6  hours  MTBOMF.  Statistical  analysis 
of  all  test  data  indicates  the  result  is  not  sufficient  to  conclude  that  reliability  has  actually  improved  since  a  Nunn-McCurdy 
review  of  the  program  in  2010.  Therefore,  test  data  currently  available  (including  early  testing  of  the  v6.0  vehicle)  do  not 
support  the  Navy’s  assertion  that  vehicle  reliability  has  improved.  Moreover,  the  current  estimate  of  RMS  reliability,  once 
all  of  the  other  components  of  the  system  are  considered,  is  no  more  than  20  hours  MTBOMF,  which  is  well-short  of  what 
is  needed  to  complete  MCM  missions  in  a  timely  fashion  and  meet  the  Navy’s  desired  mission  timelines. 

The  results  of  combined  DT/integrated  testing  completed  in  FY14  continued  to  show  that  the  RMS’s  AN/AQS-20A 
sensor  does  not  meet  Navy  requirements  for  contact  depth  localization  accuracy  (the  difference  in  depth  between  reported 
contact  position  and  ground  truth  target  position)  or  false  classification  density  (number  of  contacts  erroneously  classified 
as  mine-like  objects  per  unit  area  searched).  The  sensor  also  continues  to  have  problems  meeting  the  Navy’s  detection 
and  classification  requirements  in  shallow  waters,  and  RMS  has  difficulty  guiding  the  sensor  over  bottom  objects  for 
identification  in  deep  water.  Because  the  first  phase  of  the  LCS  IOT&E  with  an  embarked  MCM  mission  package  was 
delayed,  the  Navy  was  afforded  more  time  to  develop  an  upgraded  sensor  and  implement  other  system  changes  that  it 
expects  will  correct  these  problems.  The  program  believes  that  the  new  sensor,  AN/AQS-20B,  will  correct  or  greatly 
mitigate  the  depth  localization  and  false  classification  problems;  however,  the  AN/AQS-20B  prototypes  received  from 
the  vendor  performed  poorly  during  acceptance  and  early  characterization  testing  and  thus  required  rework.  Testing  will 
continue  in  FY15. 

Warfighter  Information  Network  -  Tactical  (WIN-T) 

WIN-T  Increment  2  is  a  two-tiered  communications 
architecture  (celestial  and  terrestrial)  that  serves  as 
the  Army’s  high-speed  and  high-capacity  tactical 
communications  network.  It  is  designed  to  provide 
reliable,  secure,  and  seamless  communications  for  units 
operating  at  theater  level  and  below.  It  supports  both 
mission  command  and  situational  awareness  through 
native  WIN-T  applications  and  existing  and  future  battle 
command  applications. 

WIN-T  has  executed  its  last  three  OTs  as  part  of  the 
Army’s  Network  Integration  Evaluations  (NIEs).  This 
includes  a  May  2012  IOT&E  and  May  2013  FOT&E  for 
which  DOT&E  prepared  an  IOT&E  report  and  operational 
assessment  report,  respectively.  A  second  FOT&E  was 
executed  in  November  2014  and  analysis  is  ongoing. 

The  NIEs  provide  access  to  a  full  brigade  equipped  with 
a  complete  set  of  battle  command  applications  to  drive 
traffic  on  the  WIN-T  network.  The  complete  brigade  is 
necessary  for  OT  to  ensure  the  WIN-T  transport  layer  can  realistically  support  the  data  needs  of  a  brigade  with  a  complete 
set  of  battle  command  applications.  While  laboratory  testing  of  these  is  possible,  it  is  difficult  to  execute,  and  DOT&E  has 


M-ATV  Point  of  Presence 


Stryker  Point  of  Presence 


1  -  Net-Centric  Waveform 

Antenna 

2  -  High-Band  Networking 

Waveform  Antenna 

M-ATV  -  Mine  Resistant  Ambush  Protected 
(MRAP)  All-Terrain  Vehicle  (M-ATV) 


M-ATV  Soldier  Network  Extension 


Tactical  Comms  Node 


Problem  Discovery  Affecting  OT&E  27 


F  Y  1  4  DOT&E  ACTIVITY  AND  OVERSIGHT 


not  yet  seen  a  DT  for  WIN-T  that  included  the  full  breadth  of  these  applications.  The  only  way  to  ensure  thorough  OT  is  to 
use  a  fully  equipped  and  trained  brigade  combat  team. 

A  concern  with  the  NIEs,  from  an  OT  perspective,  is  their  inherent  schedule-driven  nature;  NIEs  are  very  complex  events, 
which  are  held  twice  each  year.  Planning  for  the  NIEs  begins  12  to  18  months  prior  to  execution  and  systems  are  inserted 
into  the  event  after  planning  has  begun.  Relevant  Army  programs  plan  their  test  schedule  around  fitting  into  the  NIE,  rather 
than  ensuring  their  system  is  truly  ready  for  test.  The  NIE  is  but  one  example  of  external  events  driving  the  OT  schedule 
vice  scheduling  tests  to  verify  fixes  implemented  to  correct  problems  observed  in  earlier  testing. 

WIN-T  has  executed  four  OT  to  date  (including  a  Limited  User  Test  executed  in  May  2009,  prior  to  the  existence  of 
NIEs).  Some  performance  problems  identified  in  the  IOT&E  have  remained  constant  throughout  WIN-T  testing.  Many 
of  these  problems  could  not  have  been  observed  during  the  WIN-T  DTs.  Sometimes  this  was  due  to  the  limited  scope  of 
the  DTs,  sometimes  because  the  observation  requires  a  representative  unit  facing  a  representative  threat  in  an  operational 
environment.  The  problems  include: 

•  Poor  performance  of  the  line-of-sight  Highband  Networking  Waveform  (HNW)  -  HNW  is  required  to  offer  27  megabits 
per  second  (Mbps)  at-the-halt  at  a  12-kilometer  distance  and  18  Mbps  on-the-move  at  a  distance  of  2  kilometers.  OT 
has  shown  the  HNW  is  not  capable  of  providing  this  capability  to  a  dispersed  brigade.  This  could  have  been  identified 
in  DT,  but  was  not  because  of  the  limited  scope  and  benign  conditions  of  DT. 

•  Poor  performance  of  the  Soldier  Network  Extension  (SNE)  -  The  SNE  is  a  company-level  vehicle  kit  that  includes  a 
satellite  transponder  and  computer  for  connection  to  the  WIN-T  network.  At  the  IOT&E  and  FOT&E,  it  had  major 
usability  and  reliability  problems  that  were  only  discoverable  in  OT.  Identification  of  these  problems  required  the 
evaluation  of  the  ability  of  representative  trained  operators  in  an  operational  setting  to  execute  their  mission. 

•  Lack  of  Network  Operations  capability  -  Outside  of  the  central  Network  Operations  and  Security  Center,  there  is  very 
limited  capability  for  the  unit  to  monitor  and  manage  the  WIN-T  network.  This  was  only  observable  in  OT.  This  would 
have  been  difficult  to  identify  in  DT  because  it  requires  an  assessment  of  a  representative  unit’s  ability  to  monitor  and 
manage  a  dispersed  network  reacting  to  a  realistic  operational  scenario. 

•  Poor  reliability  -  The  WIN-T  Increment  2  configuration  items  were  not  reliable.  They  did  not  meet  the  Army’s 
requirements  or  serve  the  needs  of  operators  and  commanders.  The  consequences  of  reliability  problems  on  the  unit’s 
ability  to  complete  its  mission  are  discoverable  in  OT  but  not  DT.  The  context  of  OT  provides  programs  and  users  the 
magnitude  of  the  mission  consequences. 

•  Cybersecurity  vulnerabilities  -  The  Army’s  brigade-level  network  has  a  significant  number  of  cyber  vulnerabilities. 
These  vulnerabilities  can  only  be  put  into  context  and  evaluated  properly  when  tested  using  a  representative  computer 
network  defense  and  threat  employed  during  OTs,  such  as  the  NIEs.  Additionally,  cybersecurity  assessments  require 
the  presence  of  the  complete  set  of  battle  command  applications  (hardware  and  software)  and  the  support  of  external 
computer  network  defense  organizations  to  create  a  representative  environment,  which  is  only  available  through  OT. 

The  WIN-T  Program  Office  has  combined  their  efforts  with  the  cooperative  and  adversarial  cybersecurity  assessment 
teams  to  identify  vulnerabilities  and  initiate  fixes. 
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RELIABILITY 

Of  the  81  total  programs  surveyed,  21  had  reliability  problems  serious  enough  to  either  negatively  affect  the  suitability 
assessment  in  an  FY14  OT  report  or  jeopardize  successful  performance  in  OT.  Programs  are  taking  corrective  actions 
throughout  the  acquisition  cycle  to  address  reliability  problems,  but  for  some  systems,  reliability  remains  a  concern  even  after 
IOT&E  or  FOT&E.  Table  13  summarizes  this  information.  (One  system  had  reliability  problems  during  a  pre-IOT&E  test 
that  will  be  tested  in  an 
upcoming  FOT&E.) 

For  the  most  part, 
programs  are  either 
delaying  OT  to  address 
reliability  and/or  are 
implementing  fixes  to 
address  reliability  prior 
to  entering  an  OT  event; 

Table  14  summarizes 
program  responses  to 
reliability  problems.  Of 
the  eight  programs  that 
re-observed  reliability 
problems  during  an 
IOT&E  or  FOT&E, 
six  implemented  fixes  to  address 
reliability  prior  to  the  OT  event. 

Similarly,  of  the  nine  programs  with 
known  reliability  problems  that 
jeopardize  successful  performance 
in  an  upcoming  OT  event,  four  have 
delayed  OT  to  address  reliability 
(Table  14).  Early  OTs  are  the  most 
likely  to  be  delayed.  All  five  of  the 
programs  with  reliability  problems 
that  have  not  delayed  an  upcoming 
OT  have,  at  a  minimum,  identified  a 
fix,  and  four  have  implemented  fixes. 

Despite  program  attempts  to  address  reliability,  some  programs  continue  to  observe  reliability  shortfalls  during  IOT&E  and 
FOT&E.  In  part,  this  reflects  the  iterative  nature  of  reliability  improvement;  programs  go  through  multiple  cycles  of  testing 
and  implementing  fixes.  Nevertheless,  a  decrease  in  reliability  problems  observed  during  IOT&E  and  FOT&E,  as  opposed 
to  earlier  phases  of  testing,  might  be  possible  with  further  improvement  in  reliability  growth  plans.  See  the  discussion  on 
reliability  in  my  introduction  to  this  Annual  Report  for  further  details. 


TABLE  14.  PROGRAM  RESPONSES  TO  RELIABILITY  PROBLEMS 

Category 

pre-IOT&E 

IOT&E 

FOT&E 

Fix  Not 
Identified 

Fix 

Documented 

Fix 

implemented 

(tested) 

Fix  Not 
Identified 

Fix 

Documented 

Fix 

implemented 

(tested) 

Fix  Not 

Identified 

Fix 

Documented 

Fix 

implemented 

(tested) 

Known  problem 
re-observations 
(IOT&E  or  FOT&E) 

-- 

-- 

-- 

2 

-- 

4(1) 

0 

-- 

2(2) 

Problems  have 
not  delayed 
upcoming  OT 

0 

i 

0 

0 

i 

1  (0) 

0 

0 

2(1) 

TABLE  13.  FY14  OT  RELIABILITY  RESULTS  BASED  ON  NUMBER  OF  PROGRAMS 

Category 

Number  of 
programs  with 
reliability  problems1 

Pre-IOT&E 

IOT&E 

FOT&E 

Known  problem 
re-observations 

9 

1 

6 

2 

New  problem  discovery 

4 

1 

2 

1 

Problems  delayed 
upcoming  OT 

4 

2 

2 

0 

Problems  have  not 
delayed  upcoming  OT 

5 

1 

2 

2 

1 .  Twenty-one  programs  had  reliability  problems  serious  enough  to  either  affect  a  suitability  assessment  or  jeopardize  the  successful 
performance  in  an  upcoming  OT  event.  1  identified  one  program  that  had  a  reliability  problem  in  a  pre-IOT&E  event,  and  this  item  will 
be  tested  in  an  upcoming  FOT&E.  This  program  contributes  both  to  the  number  of  programs  that  have  conducted  an  OT  and  to  the 
number  programs  with  upcoming  OT  events.  Thus,  the  sum  of  the  number  of  programs  is  22. 
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PROGRESS  UPDATES  ON  DISCOVERIES  REPORTED  IN  THE  FY1 3  DOT&E  ANNUAL  REPORT 

In  FY13, 1  identified  12  systems  that  had  significant  problems  in  IOT&E  that  should  have  been  discovered  and  resolved  prior  to 
commencement  of  OT.  They  are  listed  in  Table  15  below. 

I  also  identified  10  programs  that 
re -observed  known  problems  in 
IOT&E,  shown  in  Table  16. 

One  of  the  programs  in  Table  16, 
the  Mission  Planning  System 
(MPS)  is  no  longer  under  oversight. 

The  status  of  the  remaining  systems 
is  shown  below. 

All  fixes  implemented  and 
demonstrated  in  OT 

•  AIM-9X  Air-to-Air  Missile 
Upgrade 

•  Global  Command  and  Control 
System  -  Joint  (GCCS  -J) 

Some  (or  all)  fixes  implemented  but  new  problems  discovered  or  known 
problems  re-observed  in  OT 

•  F-15E  Radar  Modernization  Program  (RMP) 

•  Manpack  Radio 

•  Joint  Battle  Command  -  Platform  (JBC-P) 

•  Miniature  Air-Launched  Decoy  (MALD)  and  MALD-Jammer  (MALD-J) 

•  P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA) 

Some  fixes  (potentially)  implemented;  Currently  in  OT  or  planning 
additional  OT 

•  AIM- 1 20  Advanced  Medium-Range  Air-to-Air  Missile  (AMRAAM) 

•  Defense  Enterprise  Accounting  and  Management  System  (DEAMS) 

•  DOD  Automated  Biometric  Identification  System  (ABIS) 

•  E-2D  Advanced  Hawkeye 

•  H-l  Upgrades  -  U.S.  Marine  Corps  Upgrade  to  AH-1Z  Attack  Helicopter  and 
UH-1Y  Utility  Helicopter 

•  Multi-Static  Active  Coherent  (MAC)  System 

•  Public  Key  Infrastructure  (PKI)  Increment  2 

•  Surveillance  Towed  Array  Sensor  System  (SURPASS)  and  Compact  Low 
Frequency  Active  (CLFA) 

•  Warfighter  Information  Network  -  Tactical  (WIN-T) 

No  Fixes  Planned 

•  Mk  54  Lightweight  Torpedo 

Not  reported  on  in  this  year’s  Annual  Report  because  no  OT  took  place  this  year 

•  Acoustic  Rapid  Commercial  Off-the-Shelf  (COTS)  Insertion  for  Sonar  AN/BQQ-10  (V)  (A-RCI)  and  the  AN/BYG-1 
Combat  Control  System 

•  Cooperative  Engagement  Capability  (CEC) 

•  Global  Broadcast  System  (GBS) 


TABLE  1 6.  FY1 3  SYSTEMS  THAT  HAD  KNOWN 
PROBLEMS  RE-OBSERVED  IN  IOT&E 

Known  Problem  Re-Observations  in  IOT&E 

AIM-120  Advanced  Medium-Range  Air-to-Air  Missile 
(AMRAAM) 

Cooperative  Engagement  Capability  (CEC) 

E-2D  Advanced  Hawkeye 

F-15E  Radar  Modernization  Program  (RMP) 

Global  Broadcast  System  (GBS) 

Global  Command  and  Control  System  -  Joint 
(GCCS-J) 

H-1  Upgrades  -  U.S.  Marine  Corps  Upgrade  to  AH-1 Z 
Attack  Helicopter  and  UH-1Y  Utility  Helicopter 

Manpack  Radio 

Mission  Planning  System  (MPS)/Joint  Mission 
Planning  Systems  -  Air  Force  (JMPS-AF) 

P-8A  Poseidon  Multi-Mission  Maritime  Aircraft 
(MMA) 

TABLE  1 5.  FY1 3  SYSTEMS  THAT  HAD  SIGNIFICANT  NEW  PROBLEM  DISCOVERY  IN  OT 

IOT&E  with  New  Problem  Discovery 

OT  other  than  IOT&E  with  New  Problem  Discovery 

AIM-9X  Air-to-Air  Missile  Upgrade 

Acoustic  Rapid  Commercial  Off-the-Shelf  Insertion 
(A-RCI)  AN  /  BQQ-10  (V)  Submarine  Sonar  System 

AIM-1 20  Advanced  Medium-Range  Air-to-Air  Missile 
(AMRAAM) 

Defense  Enterprise  Accounting  and  Management 
System  (DEAMS) 

Joint  Battle  Command  -  Platform  (JBC-P) 

DOD  Automated  Biometric  Identification  System 
(ABIS) 

Miniature  Air  Launched  Decoy  (MALD)  and 

MALD -Jammer  (MALD-J) 

Mk  54  Lightweight  Torpedo 

Multi-Static  Active  Coherent  (MAC)  System 

Public  Key  Infrastructure  (PKI) 

SurveillanceTowed  Array  Sensor  System  (SURTASS) 
and  Compact  Low  Frequency  Active  (CLFA) 

Warfighter  Information  Network -Tactical  (WIN-T) 
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In  FY13, 1  also  identified  16  systems  that  had  significant  issues  in  early  testing  that  should  be  corrected  before  IOT&E.  They 
are  listed  in  Table  17. 

The  following  provides  an  update 
on  the  progress  these  systems 
made  in  implementing  fixes  to 
those  problems.  Two  of  these 
programs  are  not  reported  on  in 
this  year’s  Annual  Report  because 
no  significant  OT  activity  occurred 
and  the  Integrated  Electronic 
Health  Record  (iEHR)  program  is 
no  longer  on  the  oversight  list. 

Fixes  tested  in  OT  -  New 
problems  discovered 

•  DOD  Automated  Biometric 
Identification  System  (ABIS) 

•  Surface  Ship  Torpedo  Defense 
(SSTD)  System:  Torpedo 
Warning  System  (TWS)  and 
Countermeasure  Anti-torpedo  (CAT) 

Fixes  tested  in  OT  -  Known  problems  re-observed 

•  CVN-78  Gerald  R.  Ford  class  Nuclear  Aircraft  Carrier 

•  Defense  Enterprise  Accounting  and  Management  System  (DEAMS) 

•  Joint  Warning  and  Reporting  Network  (JWARN) 

Fixes  tested  in  OT  -  Both  new  problems  discovered  and  known  problems  re-observed 

•  Manpack  Radio 

•  Q-53  Counterfire  Target  Acquisition  Radar  System 

Upcoming  testing  with  no  problems  identified 

•  Rifleman  Radio  and  Nett  Warrior 

•  Integrated  Defensive  Electronic  Countermeasures  (IDECM) 

Upcoming  testing  with  problems  identified 

•  LHA-6  New  Amphibious  Assault  Ship 

•  Littoral  Combat  Ship  (LCS) 

•  Public  Key  Infrastructure  (PKI) 

•  RQ-4B  Global  Hawk  High- Altitude  Long-Endurance  Unmanned  Aerial  System  (UAS) 

•  M109  Family  of  Vehicles  (FoV)  Paladin  Integrated  Management  (PIM) 

Not  reported  on  in  this  year’s  Annual  Report  because  no  OT  took  place  this  year 

•  Next  Generation  Diagnostic  System  (NGDS) 


TABLE  1 7.  FY1 3  SYSTEMS  THAT  HAD  SIGNIFICANT  ISSUES  IN  EARLYTESTING 

CVN-78  Gerald  R.  Ford  class  Nuclear  Aircraft  Carrier 

LHA-6  New  Amphibious  Assault  Ship 

Defense  Enterprise  Accounting  and  Management 
System  (DEAMS) 

Littoral  Combat  Ship  (LCS) 

DOD  Automated  Biometric  Identification  System 
(ABIS) 

Ml  09  Family  of  Vehicles  (FoV)  Paladin  Integrated 
Management  (PIM) 

Manpack  Radio 

Next  Generation  Diagnostic  System  (NGDS) 

Rifleman  Radio  and  Nett  Warrior 

Public  Key  Infrastructure  (PKI) 

Integrated  Defensive  Electronic  Countermeasures 
(IDECM) 

Q-53  Counterfire  Target  Acquisition  Radar  System 

Integrated  Electronic  Health  Record  (iEHR) 

RQ-4B  Global  Hawk  High-Altitude  Long-Endurance 
Unmanned  Aerial  System  (UAS) 

Joint  Warning  and  Reporting  Network  (JWARN) 

Surface  Ship  Torpedo  Defense  (SSTD)  System: 
Torpedo  Warning  System  (TWS)  and  Countermeasure 
Anti-torpedo  Torpedo  (CAT) 

Problem  Discovery  Affecting  OT&E 


31 


F  Y  1  4  DOT&E  ACTIVITY  AND  OVERSIGHT 


32 


DOD  Programs 


DOD  Programs 


F  Y  1  4  DOD  PROGRAMS 


Defense  Medical  Information  Exchange  (DMIX) 


Executive  Summary 

•  In  January  2014,  the  USD(AT&L)  directed  the  Program 
Executive  Officer  (PEO)  for  DOD  Healthcare  Management 
Systems  (DHMS)  to  align  all  health  data  and  interoperability 
programs,  projects,  and  initiatives  between  DOD  and  external 
organizations  under  a  single  program  and  develop  a  Health 
Data  Sharing  and  Interoperability  Roadmap  that  includes 

an  acquisition  and  technical  strategy  based  on  functional 
requirements. 

•  PEO  DHMS  created  an  Automated  Information  System  called 
Defense  Medical  Information  Exchange  (DMIX).  The  DMIX 
program  provides  the  infrastructure  and  services  to  support  the 
integrated  sharing  of  standardized  health  data  among  the  DOD 
Healthcare  Management  System  Modernization  (DHMSM) 
system,  DOD  legacy  systems,  the  Department  of  Veterans 
Affairs  (VA),  other  Federal  agencies,  and  private  sector 
healthcare  providers. 

•  The  DOD  is  acquiring  DHMSM  to  replace  DOD  legacy 
healthcare  systems  including  the  Armed  Forces  Health 
Longitudinal  Technology  Application  ( AHLTA),  the 
Composite  Health  Care  System,  and  the  Electronic  Health 
Record  (EHR)  component  of  the  Theater  Medical  Information 
Program  -  Joint  program.  Together,  DHMSM  and  DMIX  are 
intended  to  modernize  the  Military  Health  System  to  enhance 
sustainability,  flexibility,  and  interoperability,  for  improved 
continuity  of  care. 

•  The  DOD  is  developing  DMIX  incrementally,  delivering 
upgrades  to  capabilities  that  have  already  been  fielded.  During 
FY14,  three  major  capabilities  were  upgraded: 

-  Data  Federation 

-  Virtual  Lifetime  Electronic  Record  (VLER) 

-  Integrated  Electronic  Health  Record  (iEHR). 

Data  Federation 

•  The  20 1 4  N ational  Defense  Authorization  Act  required 
that  all  healthcare  data  contained  in  the  DOD’s  AHLTA 
and  the  VA’s  Veterans  Health  Information  Systems  and 
Technology  Architecture  (VistA)  systems  be  computable 
in  real  time  and  in  compliance  with  national  standards 
by  October  1,  2014.  The  DOD  and  VA  created  clinical 
terminology  maps  to  associate  health  data  from  AHLTA  and 
VistA  to  a  common  set  of  terms  based  upon  the  Health  Data 
Dictionary.  The  DOD  and  VA  delivered  terminology  maps 
for  seven  clinical  domains  in  November  2013  to  support 
Data  Federation  Release  0  (DF  R0).  The  DMIX  Program 
Management  Office  (PMO)  loaded  the  clinical  terminology 
maps  into  a  multiple  mapping  table  in  DF  R0  to  associate 
over  64,000  terms  across  the  7  clinical  domains.  The 
DMIX  PMO  intended  for  subsequent  DMIX  DF  releases 
to  add  terminology  maps  for  21  additional  clinical  data 
domains. 


AHLTA 


Med-COI 

JLVApp  Svr 
j  Meadows 


AHLTA  -  Armed  Forces  Health  Longitudinal  Technology  Application 

BHIE  -  Bidirectional  Health  Information  Exchange 

CHDR  -  Clinical  Data  Reposltory/Health  Data  Repository 

DHA  -  Defense  Health  Agency 

DoD  -  Department  of  Defense 

JLV  App  Svr  -  Joint  Legacy  Viewer  Application  Server 
Med-COI  -  Medical  Community  of  Interest 
VA  -  Veterans  Affairs 

VLER  -  Virtual  Lifetime  Electronic  Record 


VA  Systems 
and  Data 
Stores 


Joint  Legacy 
Viewer 


•  The  DOD  and  VA  were  unable  to  deliver  additional 
terminology  maps  to  support  the  testing  and  fielding  of  DF 
R1  in  September  2014.  Although  no  maps  were  delivered, 
the  DMIX  PMO  developed  DF  R1  to  allow  users  to  view 
unmapped  patient  data  for  an  additional  eight  clinical 
domains.  DF  R1  included  updates  to  both  the  Joint  Legacy 
Viewer  (JLV)  and  the  Bi-  directional  Health  Information 
Exchange  (BHIE)  DOD  Adapter.  The  DMIX  PMO 
conducted  DF  R1  system  integration  testing,  validating  the 
ability  to  view  both  mapped  and  unmapped  patient  data 
within  15  clinical  domains. 

•  A  DF  R0  operational  assessment  was  scheduled  for 
January  2014,  but  the  PEO  DHMS  did  not  support 
operational  testing  of  this  release;  therefore,  none  was 
conducted. 

•  Using  data  from  live  operations,  DOT&E  determined  that 
DF  RO’s  operational  availability  was  53.19  percent,  which 
was  unacceptably  low.  The  types  of  failures  and  frequency 
of  occurrence  suggested  systemic  problems  in  the  DMIX 
architecture  and  supporting  network  infrastructure. 

•  Operational  testing  of  DF  R1  is  planned  for  2QFY15. 

Virtual  Lifetime  Electronic  Record  (VLER) 

•  The  DMIX  PMO  conducted  system  integration  testing  of 
VLER  v2. 0.2.0  in  September  and  October  2014.  The  Army 
Test  and  Evaluation  Command  (ATEC)  and  United  States 
Army  Medical  Department  Board  (USAMEDDBD)  will 
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test  VLER  v2. 0.2.0  as  part  of  DMIX  R2  operational  testing 
in  3QFY15. 

Integrated  Electronic  Health  Record  (iEHR) 

•  In  January  2014,  USD(AT&L)  directed  PEO  DHMS 
to:  (1)  transition  all  relevant  iEHR  Increment  2  clinical 
application  requirements  to  the  DHMSM  program; 

(2)  complete  iEHR  Increment  1  Milestone  B  requirements 
including  Context  Management,  Single  Sign-On  (SSO), 
and  Roaming  capabilities;  and  (3)  conduct  an  operational 
assessment  of  iEHR  Increment  1  in  coordination  with 
DOT&E. 

•  Prior  to  being  realigned  under  the  DHMSM  and  DMIX 
programs,  iEHR  Increment  1  was  intended  to  provide 
59,000  users  at  16  sites  access  to  DOD  and  VA  patient 
data.  However,  the  DOD  recently  decided  to  limit  iEHR 
Increment  1  fielding  to  only  the  Captain  James  A.  Lovell 
Federal  Health  Care  Center  (JALFHCC),  where  the  system 
is  currently  deployed. 

•  An  operational  assessment  of  iEHR,  conducted  by 
USAMEDDBD  at  JALFHCC  in  April  2014,  demonstrated 
that  users  considered  iEHR  Increment  1  Context 
Management,  SSO,  and  Roaming  capabilities  helpful  when 
they  were  available.  However,  iEHR  Increment  1  did  not 
provide  these  capabilities  reliably.  Four  of  the  66  users 
surveyed  reported  that  Context  Management  sometimes 
displayed  data  corresponding  to  the  wrong  patient,  creating 
a  potential  risk  to  patient  safety.  JALFHCC  and  PEO 
DHMS  took  immediate  action  to  protect  patient  safety. 

•  In  J  uly  20 1 4,  U SD( AT &L)  approved  a  limited  fielding 
for  iEHR  Increment  1  to  JALFHCC  and  directed  that: 

(1)  Context  Management,  SSO,  and  Roaming  capabilities 
be  part  of  the  DHMSM  requirements;  (2)  full  deployment 
of  iEHR  Increment  1  to  additional  sites  was  no  longer 
required;  and  (3)  a  follow-on  operational  assessment  of 
iEHR  Increment  1  would  be  conducted  at  JALFHCC  to 
evaluate  system  effectiveness  following  DMIX  PMO 
corrective  actions. 

•  ATEC  and  USAMEDDBD  conducted  a  follow-on 
operational  test  in  September  2014  after  the  DMIX  PMO 
made  system  and  network  improvements  that  showed 
improved  system  availability. 

-  A  small  subset  of  users  experienced  the  Context 
Management  toolbar  disappearing  or  turning  black 
preventing  use  of  the  capability. 

-  No  patient  safety  deficiencies  were  observed. 

-  Cybersecurity  testing  could  not  be  scheduled  during  the 
operational  test  but  is  planned  for  1-2QFY15. 

•  In  November  2014,  PEO  DHMS  approved  iEHR 
Increment  1  Full  Deployment  to  JALFHCC  in  coordination 
with  the  Assistant  Secretary  of  Defense,  Health  Affairs. 

System 

•  The  DMIX  program  provides  the  infrastructure  and  services 

to  support  the  integrated  sharing  of  standardized  health  data 

among  the  DOD’s  DHMSM,  DOD  legacy  systems,  VA,  other 

Federal  agencies,  and  private  sector  healthcare  providers. 


DHMSM  will  replace  DOD  legacy  healthcare  systems 
including  AHLTA,  the  Composite  Health  Care  System,  and 
the  EHR  component  of  the  Theater  Medical  Information 
Program  -  Joint  program.  Together,  DHMSM  and  DMIX  are 
intended  to  modernize  the  Military  Health  System  to  enhance 
sustainability,  flexibility,  and  interoperability,  for  improved 
continuity  of  care. 

•  DOD  is  developing  DMIX  incrementally,  delivering  upgrades 
to  capabilities  that  have  already  been  fielded. 

-  Data  Federation  is  designed  to  advance  DOD  and  VA 
interoperability  by  providing  standardized  VA  and 
DOD  health  data  through  mapping  to  standard  medical 
terminology  using  the  JLV  browser,  which  presents 
aggregated  patient  data  from  DOD  and  VA  healthcare 
systems.  The  JLV  provides  an  integrated  read-only, 
chronological  view  of  health  data  from  DOD  and  VAEHR 
systems,  eliminating  the  need  for  VA  or  DOD  clinicians 

to  access  separate  viewers  to  obtain  real-time  patient 
information. 

-  The  VLER  provides  views  of  a  patient’s  medical  history 
and  clinical  visits  in  the  outpatient  environment  within 
DOD  medical  facilities.  The  VLER  also  provides  the 
ability  to  both  retrieve  and  share  medical  documentation 
with  external  partners,  such  as  the  VA  and  other  Federal 
and  commercial  institutions. 

-  The  BHIE  enables  the  VA  to  access  clinical  data  from 
multiple  DOD  and  VA  systems  using  the  BHIE  DOD 
Adapter,  BHIE  Share,  and  Clinical  Data  Repository/Health 
Data  Repository.  The  Clinical  Data  Repository/Health 
Data  Repository  enables  bi-directional  exchange  of 
outpatient  pharmacy  and  medication  allergy  data  for 
checking  drug-to-drug  and  drug-to-allergy  interaction. 

-  The  Medical  Community  of  Interest  system  facilitates 
seamless  connectivity  between  DOD  and  VA  healthcare 
providers,  applications,  and  data  via  a  shared,  secure 
network  capability. 

-  The  iEHR  SSO  and  Context  Management  system 
capabilities  are  intended  to  automate  user  log-on 
to  all  published  applications  via  a  Common  Access 
Card.  Context  Management  allows  users  to  enter  a 
patient  once  in  a  Context  Management  Toolbar  or 
Context  Management-enabled  application  and  the  same 
patient  will  automatically  populate  in  other  Context 
Management-enabled  applications.  iEHR  provides 

a  Roaming  capability  to  allow  users  to  access  their 
information  from  multiple  devices.  It  maintains  persistent 
virtual  desktops  for  each  user,  allowing  providers  to 
continue  viewing  and  updating  patient  records  across 
multiple  end-user  devices. 

Mission 

The  DOD,  VA,  other  Federal  agencies,  and  private  sector 

providers  will  use  the  DMIX  infrastructure  and  services  to: 

•  Support  an  integrated  sharing  of  standardized  health  data 

•  Securely  and  reliably  exchange  standardized  electronic  health 
data  with  all  partners 
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Major  Contractors 

•  Data  Federation/JLV:  Hawaii  Resource  Group  -  Honolulu, 
Hawaii 

•  Data  Federation  (Enterprise  Service  Bus/Service  Oriented 
Architecture):  Harris  -  Leesburg,  Virginia 

•  Test  Support:  Deloitte  -  Falls  Church,  Virginia 


•  JALFHCC  SSO  and  Context  Management:  General  Dynamics 
Information  Technology  -  Fairfax,  Virginia 

•  Program  Manager/VLER  support:  Technatomy  -  Fairfax, 
Virginia 


Activity 

•  PEO  DHMS  consolidated  all  health  data  sharing  and 
interoperability  programs,  projects,  and  initiatives  under  a 
automated  information  system  called  DMIX.  DOT&E  placed 
DMIX  on  test  and  evaluation  oversight  on  February  18,  2014. 
The  following  sections  discuss  testing  of  the  DMIX 
capabilities. 

Data  Federation 

•  The  20 1 4  N ational  Defense  Authorization  Act  required 
all  DOD  and  VA  healthcare  data  to  be  computable  in 
real  time  and  in  compliance  with  national  standards  by 
October  1,  2014.  The  DOD  and  VA  delivered  terminology 
maps  for  seven  clinical  domains  in  November  2013  to 
support  DF  R0. 

•  The  DMIX  PMO  conducted  DF  R0  system  integration 
testing  from  November  through  December  2013. 

•  On  December  18,  2013,  the  PEO  DHMS  approved  fielding 
of  DF  R0,  including  JLV  upgrades,  to  nine  VA  and  DOD 
sites  where  JLV  had  been  deployed  previously. 

•  DOT&E  worked  with  ATEC  and  USAMEDDBD  during 
2013  to  plan  a  DF  R0  operational  assessment.  The 
operational  assessment  was  scheduled  for  January  2014,  but 
the  PEO  DHMS  did  not  support  operational  testing  of  this 
release;  therefore,  none  was  conducted. 

•  DOT&E  received  log  files  from  the  systems  on  which 
DF  R0  was  operating  in  January  and  February  2014 
to  assess  the  availability  of  the  network  and  software 
infrastructure  of  DF  R0  following  its  deployment. 

•  From  July  through  August  2014,  the  DMIX  PMO  conducted 
system  integration  testing  of  the  next  release,  DF  Rl. 

•  From  August  through  September  20 1 4,  the  DMIX  PMO 
conducted  capacity  testing  of  JLV  v2.2.0.2  and  BHIE  DOD 
Adapter  vl.0.1.7.2  at  the  Richmond  Development  and  Test 
Center  (DTC). 

•  On  September  12,  2014,  PEO  DHMS  approved  fielding  of 
DF  Rl,  supporting  an  expanded  DOD  user  base  of  no  more 
than  1,000  users  at  68  sites. 

VLER 

•  In  September  and  October  2014,  the  DMIX  PMO 
conducted  system  integration  testing  of  VLER  v2. 0.2.0  at 
the  Richmond  DTC.  VLER  v2. 0.2.0  provides  software  and 
hardware  upgrades  to  VLER  v2.0.1.3,  the  currently  fielded 
version. 

•  ATEC  and  USAMEDDBD  will  test  VLER  v2.0.2.0  as  part 
of  DMIX  R2  operational  testing  in  3QFY15 


iEHR 

•  USAMEDDBD  conducted  an  operational  assessment  of 
iEHR  Increment  1  at  JALFHCC  in  April  2014. 

•  USAMEDDBD  conducted  a  follow-on  operational  test  of 
iEHR  Increment  1  at  JALFHCC  in  September  2014. 

Assessment 

Data  Federation 

•  The  DMIX  PMO  originally  scheduled  the  DF  R0  system 
integration  test  from  November  11  -  22,  2013,  but  extended 
the  schedule  to  correct  defects,  overcome  environment 
outages,  and  test  additional  requirements.  The  DMIX 
PMO  created  20  patient  records  along  with  medical  history 
to  exercise  terminology  mappings  within  the  7  clinical 
domains.  A  more  robust  set  of  patient  data  was  created 

for  testing  at  the  Richmond  DTC,  but  could  not  be  used 
because  the  site  was  not  yet  fully  operational.  During 
testing,  four  software  patches  were  applied  to  the  JLV  and 
two  software  patches  were  applied  to  the  BHIE  DOD 
Adapter.  After  discovered  defects  were  corrected,  all 
tested  clinical  data  were  queried,  retrieved,  and  displayed 
correctly  across  all  domains,  except  for  the  immunization 
domain,  where  4  of  32  trials  had  duplicate  entries  due  to  an 
open  BHIE  DOD  Adapter  software  defect. 

•  DOT&E  determined  that  DF  RO’s  operational  availability 
was  53.19  percent,  which  was  unacceptably  low.  The  types 
of  failures  and  frequency  of  occurrences  suggest  systemic 
problems  in  the  DMIX  architecture  and  supporting  network 
infrastructure.  User  confidence  in  DF  R0  is  expected 

to  be  negatively  affected  by  low  system  availability  and 
inadequate  outage  notification  procedures.  The  DMIX 
PMO  intends  to  replace  or  improve  these  systems  under  the 
DMIX  program  as  time  and  resources  pennit. 

•  The  DF  Rl  system  integration  test  was  expected  to  include 
8  additional  terminology  maps,  for  a  total  of  1 5  of  the 
required  28  maps.  However,  the  DOD  and  VA  were  unable 
to  deliver  any  additional  maps  in  time  to  support  testing. 
The  DMIX  PMO  had  completed  enhancements  to  JLV  and 
the  BHIE  DOD  Adapter  to  allow  users  to  view  unmapped 
patient  data  for  these  additional  eight  clinical  domains. 

The  DMIX  PMO  validated  the  integration  and  display  of 
patient  data  (both  mapped  and  unmapped)  within  the  15 
clinical  domains.  Test  metrics  included  JLV’s  ability  to 
display  a  complete  and  accurate  record,  by  validating  that 
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displayed  clinical  data  matched  those  in  the  database.  The 
DMIX  PMO  used  a  larger  data  set  of  284  patient  records, 
as  compared  to  20  patient  records  used  in  DF  R0  testing, 
resulting  in  greater  coverage  of  mapped  clinical  data. 

During  the  test  event,  two  software  patches  were  applied  to 
both  the  JLV  and  BH1E  DOD  Adapter  to  fix  high-severity 
defects.  After  the  defects  were  corrected,  all  tested  clinical 
data  were  queried,  retrieved,  and  displayed  correctly. 

•  During  capacity  testing  of  the  JLV  and  BH1E  DOD  Adapter, 
the  DMIX  PMO  encountered  schedule  delays  due  to 
system  and  DTC  environment  problems.  The  majority  of 
tests  executed  during  capacity  testing  were  unsuccessful, 

so  the  DMIX  PMO  performed  hardware  and  software 
upgrades  to  JLV  and  BH1E  DOD  Adapter  to  improve 
system  performance.  Following  system  upgrades,  JLV 
v2.2.0.2  and  BLUE  DOD  Adapter  vl.0.1.7.2  could  support 
300  concurrent  users,  exceeding  the  250-user  requirement. 
DOT&E  analysis  of  JLV  user  activity  through  July  2014 
showed  limited  use  of  JLV,  resulting  in  a  concurrent  user 
load  well  below  the  required  capacity. 

•  Subsequent  DMIX  DF  releases  were  planned  to  add 
terminology  maps  for  21  additional  clinical  data  domains, 
but  the  DOD  and  VA  were  unable  to  deliver  any  additional 
maps  to  support  the  testing  and  fielding  of  DF  R1  in 
September  2014.  It  is  unclear  when  either  the  DOD  or 
VA  will  provide  additional  maps  in  support  of  the  DMIX 
program. 

•  Operational  testing  of  DF  R1  is  planned  for  2QFY15. 

ATEC  and  USAMEDDBD  will  assess  the  accuracy  and 
completeness  of  mapped  terminology,  as  well  as  the 
procedures  and  tools  to  maintain  terminology  maps,  as  part 
of  DMIX  R2  operational  testing. 

VLER 

•  No  operational  test  data  are  available  to  assess  VLER. 

iEHR 

•  The  operational  assessment,  conducted  in  April  2014, 
showed  that  most  JALFHCC  users  considered  iEHR 
Increment  1  Context  Management,  SSO,  and  Roaming 
capabilities  helpful  when  they  were  available.  However, 
iEHR  Increment  ldid  not  provide  these  capabilities  reliably. 


Four  of  the  66  users  surveyed  reported  that  Context 
Management  sometimes  displayed  data  corresponding  to 
the  wrong  patient,  creating  a  potential  risk  to  patient  safety. 
JALFHCC  alerted  all  personal  of  the  potential  patient  safety 
problem  and  provided  mitigation  instructions.  PEO  DHMS 
deployed  developer  and  test  teams  to  support  JALFHCC, 
but  the  teams  were  unable  to  duplicate  the  problem. 

Because  the  problem  could  not  be  re-created  in  a  laboratory 
test  environment,  it  requires  operational  testing  to  resolve. 

•  User  inability  to  connect  to  the  Application  Virtualization 
Hosting  Environment  was  a  pervasive  problem,  resulting 
in  a  loss  of  Context  Management,  SSO,  and  Roaming 
capabilities.  As  a  workaround,  users  kept  trying  to 
connect  until  the  system  worked.  This  cumbersome  and 
time-consuming  process  discouraged  users  from  using  the 
system. 

•  The  follow-on  operational  test,  which  ATEC  and 
USAMEDDBD  conducted  after  system  and  network 
improvements,  showed  improved  system  availability. 
However,  a  small  subset  of  users  experienced  the 
Context  Management  toolbar  disappearing  or  turning 
black  preventing  use  of  the  capability.  No  patient  safety 
issues  were  observed.  Cybersecurity  testing  could  not  be 
scheduled  during  the  test  and  is  planned  for  1-2QFY15. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 

report  for  this  program. 

•  FY 14  Recommendations. 

1.  The  DOD  and  VA  should  accelerate  clinical  terminology 
mapping  efforts;  or  if  not  feasible,  pursue  an  alternate 
approach  to  exchange  healthcare  data  in  real  time  between 
the  departments. 

2.  The  DMIX  PMO  should  correct  the  iEHR  Increment  1 
toolbar  defect  to  ensure  the  system  is  available  to  all  users. 

3.  The  DMIX  PMO  should  conduct  cybersecurity  testing  of 
iEHR  Increment  1 . 

4.  ATEC  and  USAMEDDBD  should  conduct  operational 
testing  of  DF  R1  and  DMIX  R2  as  planned. 
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Department  of  Defense  (DOD)  Teleport 


Executive  Summary 

•  The  Defense  Information  Systems  Agency  (DISA)  is 
developing  DOD  Teleport  Generation  3.  Generation  3  is 
comprised  of  three  satellite  gateway  improvements  separated 
into  phases: 

-  Phase  1  is  intended  to  upgrade  the  Extremely  High 
Frequency  (EHF)  gateway  capabilities  used  to 
communicate  with  Advanced  Extremely  High  Frequency 
(AEHF)  and  Milstar  satellites. 

-  Phase  2  is  designed  to  upgrade  X-  and  Ka-band  capabilities 
used  to  communicate  with  the  Defense  Satellite 
Communications  System  (DSCS)  and  Wideband  Global 
Satellite  Communications  (SATCOM)  (WGS)  system. 

-  Phase  3  is  intended  to  provide  Mobile  User  Objective 
System  (MUOS)  users  interoperability  with  legacy 
Ultra-High  Frequency  (UHF)  users. 

•  The  Joint  Interoperability  Test  Command  (JITC)  conducted 
the  operational  test  of  the  Generation  3  Phase  1  (G3P1) 
capability  at  the  Northwest  Teleport  in  Chesapeake,  Virginia, 
from  July  21  through  September  19,  2014.  Deployed 
operational  users  from  the  Army,  Marine  Corps,  Navy,  and 
National  Guard  communicated  over  Milstar  and  AEHF 
satellites  at  data  rates  up  to  8.192  Megabits  per  second  (Mbps) 
using  Defense  Information  Systems  Network  (DISN)  services. 
G3P1  developmental  test  results  and  preliminary  analysis  of 
the  operational  test  data  suggest  the  DOD  Teleport  system 

is  capable  of  interoperating  with  deployed  EHF  users  to 
provide  end-to-end  information  exchanges  for  data  and  voice 
communications. 

•  JITC  conducted  the  operational  test  of  the  Generation  3 
Phase  2  (G3P2)  capability  at  the  Northwest  Teleport  from 
July  21  through  October  10,  2014.  Deployed  operational  users 
from  the  Army,  Air  Force,  and  Navy  communicated  over  WGS 
and  DSCS  satellites  using  DISN  services,  including  secure  and 
non-secure  video  teleconferencing.  Preliminary  analysis  of 
the  G3P2  operational  test  data  suggests  that  the  DOD  Teleport 
is  capable  of  interoperating  with  deployed  X-  and  Ka-band 
users  to  provide  end-to-end  information  exchanges  for  data, 
voice,  and  video  communications. 

•  The  Generation  3  Phase  3  (G3P3)  capability  has  been  delayed 
due  to  MUOS  waveform  reliability  problems  currently  being 
addressed  by  the  MUOS  program.  JITC  projects  the  G3P3 
operational  test  will  take  place  2QFY16  after  DISA  completes 
developmental  testing. 

System 

DOD  Teleport  sites  are  globally-distributed  SATCOM  facilities. 

The  system  has  six  core  Teleport  facilities  located  in  Virginia, 

Germany,  Italy,  Japan,  Hawaii,  and  California,  and  three 

secondary  facilities  located  in  Bahrain,  Australia  (future),  and 

Guam.  Teleport  sites  consist  of  four  segments: 


•  The  radio  frequency  segment  consists  of  SATCOM  earth 
terminals  that  operate  in  UHF,  X,  C,  Ku,  Ka,  and  EHF 
frequency  bands.  The  terminals  provide  radio  frequency  links 
between  the  Teleport  site  and  the  deployed  user  SATCOM 
terminal  via  military  or  commercial  satellites. 

•  The  base-band  segment  includes  encryption,  switching, 
multiplexing,  and  routing  functions  for  connecting  data 
streams  or  packetized  data  to  the  terrestrial  DISN. 

•  The  network  services  segment  provides  connectivity  to 
the  DISN  long-haul  networks  and  other  internet-working 
functions  necessary  to  meet  the  user’s  requirements. 

•  The  management  control  segment  provides  centralized 
monitoring  and  control  of  Teleport  base-band  hardware,  earth 
terminal  hardware,  transmission  security,  and  test  equipment. 

•  Teleport  provides  deployed  forces  access  to  standard  fixed 
gateways  from  anywhere  in  the  world  for  all  six  DISN 
services: 

-  Secret  Internet  Protocol  Router  Network  (SIPRNET) 

-  Non-secure  Internet  Protocol  Router  Network  (N1PRNET) 

-  Defense  Red  Switch  Network  (DRSN) 

-  Defense  Switched  Network  (DSN) 

-  Video  Teleconference  (VTC) 

-  Joint  Worldwide  Intelligence  Communications  System 
(JWICS) 

Mission 

Combatant  Commanders,  Services,  and  deployed  operational 
forces  use  DOD  Teleport  systems  in  all  phases  of  conflict  to  gain 
access  to  worldwide  military  and  commercial  SATCOM  services. 

Major  Contractor 

Government  Integrator:  DISA 
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Activity 

•  Teleport  Generation  3  is  comprised  of  three  satellite  gateway 
improvements  separated  into  phases: 

-  G3P 1  is  intended  to  upgrade  the  existing  EHF  gateway 
capability  with  Navy  Multiband  Terminals  that  provide 
data  rates  up  to  8.192  Mbps. 

-  G3P2  is  designed  to  upgrade  the  existing  X-  and  Ka-band 
gateway  capabilities  with  Modernization  of  Enterprise 
Terminals,  which  can  simultaneously  access  both  X-  and 
Ka-band  communications  on  the  WGS  satellites. 

-  G3P3  is  intended  to  provide  interoperability  between 
MUOS  users  and  legacy  UHF  users. 

•  DISA  installed  and  integrated  three  Navy  Multiband  Terminals 
at  the  Northwest  Teleport  site  in  Chesapeake,  Virginia,  and 
conducted  G3P1  developmental  testing  in  January  2014. 

•  DISA  and  J1TC  conducted  the  G3P1  integrated  test  from 
April  21  through  May  16,  2014,  at  the  Northwest  Teleport 
with  deployed  Army  and  Navy  users  communicating  over  the 
AEHF  satellite  system  accessing  DISN  services,  including 
secure  and  non-secure  data  and  voice  communications. 

•  J1TC  conducted  the  operational  test  of  the  G3P1 
capability  at  the  Northwest  Teleport  from  July  21  through 
September  19,  2014.  Deployed  operational  users  from 
the  Army,  Marine  Corps,  Navy,  and  National  Guard 
communicated  over  Milstar  and  AEHF  satellites  at  data  rates 
up  to  8.192  Mbps  accessing  DISN  services. 

•  DISA  installed  and  integrated  a  Modernization  of  Enterprise 
Terminal  at  the  Northwest  Teleport  site  and  conducted  G3P2 
developmental  testing  from  April  28  through  May  16,  2014, 
communicating  with  deployed  Marine  Corps  and  Air  Force 
users  over  WGS  satellites  accessing  DISN  services. 

•  JITC  conducted  the  operational  test  of  the  G3P2 
capability  at  the  Northwest  Teleport  from  July  21  through 
October  10,  2014.  Deployed  operational  users  from  the  Army, 
Air  Force,  and  Navy  communicated  over  WGS  and  DSCS 
satellites  accessing  DISN  services,  including  secure  and 
non-secure  video  teleconferencing. 

•  DISA  has  installed  a  test  suite  of  the  MUOS  to  Legacy 
UHF  capability  at  the  DOD  Teleport  Test  Lab  within 
the  Army’s  Joint  Satellite  Engineering  Center,  at 
Aberdeen  Proving  Ground,  Maryland.  Site  acceptance  testing 
at  the  Northwest  Teleport  is  planned  for  April  2015  with  an 
operational  test  event  projected  for  2016. 

•  JITC  conducted  G3P1  and  G3P2  testing  in  accordance  with  a 
DOT&E-approved  Test  and  Evaluation  Master  Plan  and  test 
plan. 

•  DOT&E  submitted  an  OT&E  Report  on  the  DOD  Teleport  in 
November  2014. 


Assessment 

•  The  DISA-conducted  G3P1  integrated  test  was  operationally 
representative  and  demonstrated  the  Teleport  capability  to 
perform  as  a  satellite  gateway  for  deployed  EHF  users  to 
connect  to  and  use  DISN  services. 

•  Although  analysis  of  the  G3P1  operational  test  data  is 
ongoing,  developmental  test  results  and  preliminary  analysis 
of  operational  test  data  suggest  that  the  DOD  Teleport 

is  capable  of  interoperating  with  deployed  EHF  users  to 
provide  end-to-end  information  exchanges  for  data  and  voice 
communications. 

•  The  DISA-conducted  G3P2  developmental  test  was 
operationally  representative,  but  provided  only  a  limited 
set  of  data.  The  DISA  program  manager  concluded  the 
developmental  test  reduced  enough  risk  to  forego  the 
integrated  test  and  proceed  directly  to  the  operational  test. 

•  Preliminary  analysis  of  the  G3P2  operational  test  data 
suggests  that  the  DOD  Teleport  is  capable  of  interoperating 
with  deployed  X-  and  Ka-band  users  to  provide  end-to-end 
information  exchanges  for  data,  voice,  and  video 
communications. 

•  A  DOT&E  recommendation  from  the  November  2009  G2P2 
MOT&E  for  DISA  to  develop  a  network  management 
system  in  coordination  with  the  Theater  Network  Operations 
Center  (TNC)  to  support  Continuity  of  Operations  remains 
unresolved.  If  communication  is  lost  between  the  TNC  and 
the  Teleport  system,  the  Teleport  operators  cannot  manage 
and  control  internet  protocol-based  deployed  users’  networks; 
therefore,  the  deployed  users’  communications  are  at  risk. 

•  The  G3P3  capability  has  been  delayed  due  to  MUOS 
waveform  reliability  problems  currently  being  addressed  by 
the  MUOS  program.  JITC  projects  the  G3P3  operational  test 
will  take  place  2QFY16  after  DISA  completes  developmental 
testing.  There  is  no  DOT&E-approved  Test  and  Evaluation 
Master  Plan  for  G3P3. 

Recommendations 

•  Status  of  Previous  Recommendations.  DISA  has  satisfactorily 
addressed  all  three  previous  recommendations. 

•  FY 14  Recommendations.  DISA  should: 

1.  Resolve  the  lien  from  the  November  2009  G2P2  MOT&E 
to  develop  a  network  management  system  in  coordination 
with  the  TNC  to  support  Continuity  of  Operations. 

2.  Update  the  Test  and  Evaluation  Master  Plan  to  support 
G3P3  testing. 
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F-35  Joint  Strike  Fighter  (JSF) 


Executive  Summary 

Test  Planning,  Activity,  and  Assessment 

•  The  program  focused  on  completing  F-35  Joint  Strike 
Fighter  (JSF)  Block  2B  development  and  flight  testing  in 
an  effort  to  provide  limited  combat  capability  to  the  fielded 
early  production  aircraft  and  to  support  the  Marine  Corps 
plans  for  declaring  Initial  Operational  Capability  (IOC)  in 
2015. 

-  The  test  centers  sustained  flight  operations  at  nearly 
the  planned  pace  through  the  end  of  November,  despite 
stoppages  and  restrictions  placed  on  the  test  fleet  of 
aircraft. 

-  Flights  sciences  testing  for  the  F-35  A  lagged  behind 
its  test  flight  and  test  point  goals  for  CY14  as  the  test 
centers  prioritized  resources  to  focus  on  Block  2B  mission 
systems  testing.  Flight  sciences  testing  for  the  F-35B 
and  F-35C  maintained  overall  test  point  productivity  by 
accomplishing  additional  test  points  for  Block  3F,  while 
lagging  behind  planned  progress  for  completing  Block  2B. 

-  Test  flights  using  the  mission  systems  aircraft  were  ahead 
of  the  plan  for  the  year,  but  test  point  productivity  for 
Block  2B  and  Block  3i  lagged  behind  the  annual  plan. 

•  In  spite  of  the  focused  effort,  the  program  was  not  able  to 
accomplish  its  goal  of  completing  Block  2B  flight  testing  by 
the  end  of  October. 

-  Slower  than  planned  progress  in  mission  systems,  weapons 
integration,  and  F-35B  flight  sciences  testing  delayed 

the  completion  of  the  testing  required  for  Block  2B  fleet 
release.  The  program  now  projects  this  to  occur  by  the  end 
of  January  2015,  instead  of  the  end  of  October  2014  as  was 
previously  planned. 

-  Restrictions  imposed  on  the  test  fleet  as  a  result  of  the 
engine  failure  in  June  reduced  test  point  availability  and 
slowed  progress  in  mission  systems  and  flight  sciences 
testing  from  July  through  November.  For  example,  the 
effect  on  mission  systems  testing  was  approximately 

17  percent  loss  of  productivity  in  accomplishing  test 
points,  from  210  points  accomplished  per  month  prior  to 
the  engine  restrictions  to  approximately  175  points  per 
month. 

-  Discoveries  of  deficiencies  continued  to  occur  in  later 
versions  of  Block  2B  software,  further  slowing  progress. 
For  example,  completion  of  weapons  delivery  accuracy 
events  lagged  the  plans  for  CY14  and  was  put  on  hold  in 
August  when  the  program  discovered  a  deficiency  in  the 
F-35  navigation  system. 

-  Through  the  end  of  November,  10  of  15  weapon  delivery 
events  had  been  completed;  all  events  were  planned  to  be 
completed  by  the  end  of  October.  However,  the  program 
must  transition  development  and  flight  test  resources  to 
Block  3  in  order  to  preserve  an  opportunity  to  complete 


the  System  Design  and  Development  phase  as  planned 
in  2018.  Block  2B  will  finish  later  than  planned,  with 
deficiencies  remaining  that  will  affect  operational  units; 
fixes  for  these  deficiencies  will  be  deferred  to  Blocks  3i 
and  3F. 

•  In  the  FY13  Annual  Report,  DOT&E  estimated  that  the 
program  would  complete  Block  2B  testing  between  May 
and  November  2015  (7  to  13  months  late),  depending  on  the 
level  of  growth  experienced,  while  assuming  the  program 
would  continue  test  point  productivity  equal  to  that  of  the 
preceding  12  months.  Since  the  end  of  October  2013,  the 
program  has  made  several  adjustments  to  reduce  the  delay 
estimated  in  the  FY13  report: 

-  In  February  2014,  while  finalizing  the  2014  annual  plan, 
the  program  consolidated  test  points  from  plans  of  earlier 
blocks  of  mission  systems  (Blocks  1A,  IB,  and  2A) 
with  those  from  the  Block  2B  test  plan  and  decided  to 
account  for  only  those  test  points  needed  for  Block  2B 
fleet  release,  eliminating  approximately  840  points.  All 
of  these  points  were  planned  to  be  accomplished  as  of  the 
DOT&E  report.  This  reduction  amounts  to  approximately 
four  months  of  testing. 
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-  Further  adjustments  to  the  baseline  number  of  test  points 
needed  for  Block  2B  fleet  release  were  made  in  June  2014, 
resulting  in  additional  reduction  of  points  planned  for  the 
year.  Although  the  program  added  points  for  new  testing 
requirements  (i.e.,  Manual  Ground  Collision  Avoidance 
System),  they  also  eliminated  points  that  were  assessed  as 
no  longer  required.  These  adjustments  resulted  in  the  net 
reduction  of  135  points. 

-  The  program  continued  to  experience  an  average 
test  point  growth  rate  throughout  CY14  higher  than 
planned  (9 1  percent  growth  experienced  through  the 
end  of  November,  45  percent  planned),  but  lower  than 
experienced  in  CY13  (124  percent). 

-  The  program  realized  a  higher  test  point  productivity  rate 
per  aircraft  in  CY14  than  in  CY13  (averaging  40  points 
per  aircraft  per  month  through  the  end  of  November, 
compared  to  35). 

-  The  program  delayed  plans  to  transition  aircraft  out  of 
the  Block  2B  configuration  to  the  Block  3i  configuration, 
allowing  more  mission  systems  test  aircraft  to  be  available 
to  contribute  to  Block  2B  testing.  At  the  time  of  this 
report,  only  AF-3  had  been  modified  to  the  Block  3i 
configuration,  among  the  six  mission  systems  test  aircraft 
assigned  to  the  Edwards  AFB  test  center,  California, 
where  the  majority  of  the  mission  systems  testing  is 
accomplished.  BF-5,  a  mission  systems  test  aircraft 
assigned  to  the  Patuxent  River  test  center,  Maryland,  was 
modified  into  the  Block  3i  configuration  in  September 
and  completed  limited  Block  3i  testing  prior  to  entering 
climatic  testing  later  in  the  month. 

Based  on  test  point  accomplishment  rates  experienced 
since  October  2013,  the  program  will  complete  Block  2B 
development  in  February  2015. 

-  This  estimate  assumes  no  further  growth  in  Block  2B 
testing  (this  is  possible  only  if  the  current  version  entering 
test  is  the  final  Block  2B  version)  and  productivity  at 

the  current  rate.  It  further  assumes  all  current  Block  2B 
mission  systems  aircraft  staying  in  the  Block  2B 
configuration  through  the  end  of  January  2015  (the 
program’s  estimated  completion  date  for  Block  2B 
development),  then  one  F-35B  and  one  F-35C  mission 
systems  test  aircraft  converting  to  Block  3i  while  the 
other  three  stay  in  the  Block  2B  configuration  until 
developmental  testing  is  complete.  Also,  the  operating 
restrictions  stemming  from  the  engine  failure  must  be 
relieved  for  the  test  aircraft  such  that  all  blocked  test 
points  are  made  available. 

-  Completion  of  Block  2B  development  by  the  end  of 
January  will,  therefore,  require  a  significant  increase  in 
test  point  productivity  and/or  elimination  of  additional  test 
points. 

In  April,  the  program  accepted  a  DOT&E  recommendation 
that  the  Block  2B  Operational  Utility  Evaluation  (OUE), 
which  was  being  planned  for  CY15,  should  not  be  conducted 
and  that  instead,  resources  should  be  focused  on  conducting 
limited  assessments  of  Block  2B  capability  and  re-allocated 


to  assist  in  the  completion  of  development  and  testing  of 
Block  3i  and  Block  3F  capabilities. 

-  This  recommendation  was  based  on  DOT&E’s  review  of 
Block  2B  progress  and  assessment  of  the  program’s  ability 
to  start  the  Block  2B  OUE  as  planned  without  creating  a 
significant  impact  to  Block  3F  development. 

-  The  Program  Office,  JSF  Operational  Test  Team,  and 
Service  representatives  then  began  working  to  “re-scope” 
use  of  operational  test  aircraft  and  operational  test 
activities  in  lieu  of  the  OUE — detailed  planning  is  still 
under  development.  The  scope  of  the  operational  test 
activities  will  be  limited  until  the  flight  restrictions  induced 
by  the  engine  failure  are  removed  from  the  operational  test 
aircraft.  Availability  of  the  operational  test  aircraft  will 
continue  to  be  affected  in  CY15  and  CY16  by  the  depot 
time  required  for  modifications. 

F-35A  Engine  Failure 

•  Asa  result  of  the  engine  failure  that  occurred  in  an  F-35A  in 
late  June,  the  program  imposed  aircraft  operating  limitations 
(AOL)  on  all  variants  of  F-35  aircraft  at  the  flight  test  centers 
and  operational/training  bases.  These  AOLs  were: 

-  Maximum  speed  of  1.6  Mach  (0.9  Mach  for  production 
aircraft  at  operational/training  bases), 

-  Maximum  g-load  of  3.2  g  for  test  aircraft  and  3.0  for 
production  aircraft, 

-  Maneuvers  limited  to  half-stick  roll  rate  and  1 8  degrees 
angle  of  attack 

-  No  rudder  input,  unless  required  for  safe  flight  (production 
aircraft  restriction  only) 

-  Note:  In  some  circumstances  during  flight  test  (but  not  in 
operational/training  aircraft),  exceedances  were  permitted 
and  testing  continued,  controlled  by  the  flight  test  team 
monitoring  the  aircraft,  on  an  aircraft-by-aircraft  basis 
(i.e.,  individual  aircraft  are  cleared  for  specific  test  points). 

•  Due  to  the  AOL,  numerous  test  points  needed  for  the 
Block  2B  fleet  release  and  Marine  Corps  IOC  were  blocked 
and  cannot  be  attempted  until  the  restrictions  are  lifted. 

-  These  test  points  include: 

■  Loads  and  buffet,  Short  Take-off  and  Vertical  Landing 
(STOVL)  envelope  expansion,  and  propulsion  testing 
for  F-35B  flight  sciences 

■  Loads  and  buffet  for  F-35A  flight  sciences  testing 

■  Manual  ground  collision  avoidance  system  testing  (for 
both  aircraft).  The  manual  ground  collision  avoidance 
system  is  a  warning  system  that  alerts  the  pilot  that  the 
state  of  aircraft  attitude  and  altitude  may  be  entering  an 
unsafe  condition  (Service  IOC  requirement). 

-  There  was  also  a  requirement  to  inspect  the  engine  with 
borescope  equipment  after  no  more  than  three  flight  hours; 
this  creates  additional  down  time  and  places  stringent 
scheduling  requirements,  which  negatively  affects  aircraft 
availability. 

■  Restrictions  for  test  aircraft  were  gradually  reduced 
between  June  and  November,  allowing  access  to  more 
test  points.  The  program  developed  a  procedure  to 
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“rub-in”  the  seal  in  the  stators  of  the  engines  in  the 
test  aircraft.  Once  this  procedure  was  accomplished, 
restrictions  were  eased  to  allow  greater  g  and  angle  of 
attack,  but  not  to  the  full  limits  of  the  planned  Block  2B 
envelope. 

■  The  program  began  installing  “pre-trenched”  stators 
(where  clearance  between  the  stator  and  rotor  has 
already  been  cut  into  the  seal  and  no  rub-in  procedure  is 
necessary)  in  the  engines  of  the  test  aircraft  in  October, 
as  they  became  available,  to  remove  the  restrictions 
associated  with  the  engine  failure.  By  the  end  of 
November,  6  of  the  18  test  aircraft  had  the  pre-trenched 
stators  installed.  The  program  plans  to  have  the  engines 
in  all  developmental  test  aircraft  modified  by  the  end 
of  February  2015.  Also,  the  borescope  inspection 
requirements  were  removed  in  November,  with  the  latest 
revision  of  the  list  of  restrictions.  However,  fielded 
production  aircraft  remained  restricted  at  the  time  of  this 
report. 

Mission  Data  Load  Development  and  Testing 

•  The  F-35  relies  on  mission  data  loads  -  which  are  a 
compilation  of  the  mission  data  files  needed  for  operation  of 
the  sensors  and  other  mission  systems  components  -  working 
in  conjunction  with  the  system  software  data  load  to  drive 
sensor  search  parameters  and  to  identify  and  correlate  sensor 
detections  of  threat  radar  signals.  The  loads  will  be  produced 
by  a  U.S.  government  lab,  the  U.S.  Reprogramming  Lab. 

-  The  first  two  mission  data  loads  support  the  Marine  Corps 
IOC,  planned  for  July  2015.  Because  the  lab  received 

its  equipment  late  from  the  major  contractor  who 
produces  the  equipment,  and  with  limited  capability,  the 
first  two  mission  data  loads  will  not  be  available  until 
November  2015. 

-  Mission  data  loads  undergo  a  three-phased  lab 
development  and  test  regimen,  followed  by  flight  test. 

The  current  plans  are  to  certify  the  first  two  mission  data 
loads  in  November  2015  after  flight  testing  occurs  between 
March  and  October  2015.  Although  this  is  later  than 
desired  by  the  program  and  the  Marine  Corps,  truncating 
the  mission  data  load  development  and  conducting  flight 
testing  early  on  a  limited  open-air  range  for  the  purpose 
of  releasing  a  mission  data  load  in  mid-2015  would  create 
significant  operational  risk  to  fielded  units,  since  the  load 
will  not  have  completed  the  planned  lab  test  regimen  and 
because  the  test  infrastructure  on  the  open-air  range  is 
capable  of  verifying  only  a  small  portion  of  the  mission 
data. 

Weapons  Integration 

•  Progress  in  weapons  integration,  in  particular  the  completion 
of  planned  Block  2B  weapon  delivery  accuracy  ( WDA) 
events,  has  been  less  in  2014  compared  to  that  planned  by  the 
program.  The  program  planned  to  complete  all  15  Block  2B 
WDA  events  by  the  end  of  October,  but  completed  only  7. 
Through  the  end  of  November,  the  program  completed  10 


Block  2B  WDA  events  and  deferred  2  to  Block  3F  testing 
due  to  deficiencies  and  limitations  in  Block  2B  capabilities. 
The  remaining  3  Block  2B  WDA  events  are  scheduled  to  be 
completed  by  the  end  of  January  2015. 

-  Multiple  deficiencies  in  mission  systems,  aircraft 
grounding,  and  subsequent  flight  restrictions  caused  by  the 
June  engine  failure  all  contributed  to  the  limited  progress. 

-  In  addition,  all  WDA  events  were  put  on  hold  in  August, 
when  a  deficiency  in  the  aircraft’s  navigation  solution  was 
discovered.  Corrections  to  the  deficiency  were  tested  and 
confirmed  in  October,  permitting  Block  2B  WDA  events  to 
restart  in  November. 

Suitability 

•  Overall  suitability  continues  to  be  less  than  desired  by 
the  Services,  and  relies  heavily  on  contractor  support 
and  unacceptable  workarounds,  but  has  shown  some 
improvement  in  CY14. 

-  Aircraft  availability  was  flat  over  most  of  the  past  year, 
maintaining  an  average  for  the  fleet  of  37  percent  for  the 
12-month  rolling  period  ending  in  September  -  consistent 
with  the  availability  reported  in  the  FY13  DOT&E 
report  of  37  percent  for  the  12-month  period  ending 

in  October  2013.  However,  the  program  reported  an 
improved  availability  in  October  2014,  reaching  an  average 
rate  of  5 1  percent  for  the  fleet  of  90  aircraft  and  breaking 
50  percent  for  the  first  time,  but  still  short  of  the  program 
objective  of  60  percent  set  for  the  end  of  CY14.  The 
bump  in  availability  in  October  brought  the  fleet  12-month 
average  to  39  percent. 

-  Measures  of  reliability  and  maintainability  that  have 
Operational  Requirements  Document  (ORD)  requirements 
have  improved  since  last  year,  but  all  nine  reliability 
measures  (three  for  each  variant)  are  still  below  program 
target  values  for  the  current  stage  of  development. 

-  The  reliability  metric  that  has  seen  the  most  improvement 
since  May  2013  is  not  an  ORD  requirement  but  a  contract 
specification  metric,  mean  flight  hour  between  failures 
scored  as  “design  controllable”  (which  are  equipment 
failures  due  to  design  flaws).  For  this  metric,  the  F-35B 
and  F-35C  are  currently  above  (better  than)  program 
target  values,  and  F-35  A  is  slightly  below  (worse  than)  the 
target  value  but  has  been  above  the  target  value  for  several 
months  over  the  last  year. 

Live  Fire  Test  and  Evaluation  (LFT&E) 

•  The  F-35  LFT&E  program  completed  two  major  live  fire  test 
series  using  an  F-35B  variant  full-scale  structural  test  article. 
Preliminary  evaluations  are  that  the  tests: 

-  Demonstrated  the  capabilities  of  multiple  structural  wing 
load  paths  and  aft  boom  structure  to  mitigate  threat-induced 
large  scale  structural  failure. 

-  Confirmed  the  expected  vulnerabilities  of  the  fuel  tank 
structure. 
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-  Demonstrated  the  expected  cascading  damage 

vulnerability  to  fuel  ingestion,  fuel  and  hydraulic  fire,  and 
hydrodynamic  ram  events. 

•  Engine  live  fire  tests  in  FY13  and  prior  live  fire  test  data 
and  analyses  demonstrated  vulnerability  to  engine  fire, 
either  caused  by  cascading  effects  or  direct  damage  to 
engine  fuel  lines  and  fueldraulic  components.  Additional 
details  and  analyses  of  the  uncontained  F135  fan  blade 
release  and  subsequent  fuel  fire  in  an  F-35A  at  Eglin  AFB 
in  June  are  needed  to  support  and  update  the  existing  engine 
vulnerability  assessment. 

•  The  program  demonstrated  performance  improvements  of 
the  redesigned  fuel  tank  ullage  inerting  system  in  the  F-35B 
ground-based  fuel  system  simulator.  Flowever,  aircraft 
ground  and  flight  tests,  designed  to  validate  the  fuel  system 
simulator  tests  and  aircraft  system  integration,  revealed 
redesign  deficiencies  that  require  further  hardware  and 
software  modifications. 

•  Lockheed  Martin  provided  test  and  analysis  results  to  resolve 
the  concern  expressed  in  FY13  for  the  potential  aircraft  loss 


due  to  ballistically-induced  shorting  of  the  270  Volt  and 
28  Volt  flight  control  electrical  systems.  Protection  on  the 
28  Volt  electrical  system  (designed  for  lightning  protection) 
provides  tolerance  to  such  a  single  ballistic  shorting  event 
and  is  unlikely  to  result  in  a  loss  of  aircraft. 

•  The  F-35  program  continues  to  make  progress  in  assessing 
the  survivability  of  the  F-35  to  unconventional  threats. 
Development  of  the  chemical  and  biological  agent  protection 
and  decontamination  systems  will  be  evaluated  in  the 
full-up  system-level  decontamination  test  planned  for  FY16. 
The  Navy  has  been  testing  the  vulnerability  of  the  F-35B 
electrical  and  mission  systems  to  electromagnetic  pulse 
(EMP),  and  plans  to  complete  this  testing  by  2QFY15. 

•  The  program  is  making  advances  in  assessing  the  lethality 
of  the  25  mm  x  137  mm  PGU-48  Frangible  Armor  Piercing 
(FAP)  round,  a  designated  round  for  the  F-35A  variant, 
and  the  PGU-32/U  Semi-Armor  Piercing  High  Explosive 
Incendiary-Tracer  (SAPHEI-T)  ammunition  currently 
designated  for  the  F-35B  and  F-35C  variants. 


Actual  versus  Planned  Test  Metrics  through  November  201 4 


TEST  FLIGHTS 

All  Testing 

Flight  Sciences 

Mission 

Systems 

All  Variants 

F-35BOnly 

F-35A  Only 

F-35C  Only 

2014  Actual 

1,268 

313 

197 

286 

472 

2014  Planned 

1,209 

296 

262 

261 

390 

Difference  from  Planned 

4.9% 

5.7% 

-24.8% 

9.6% 

21.0% 

Cumulative  Actual 

5,046 

1,648 

1,194 

944 

1,260 

Cumulative  Planned 

4,674 

1,471 

1,205 

894 

1,104 

Difference  from  Planned 

8.0% 

12.0% 

-0.9% 

5.6% 

14.1% 

TEST  POINTS 


All  Testing 

Flight  Sciences' 

Mission  Systems 

All  Variants 

F-35BOnly 

F-35AOnly 

F-35COnly 

Block  2B2 

Block  3i 

Block  3F 

Other 

2B 

3F 

2B 

3F 

2B 

3F 

2014  Baseline  Accomplished 

7,055 

1,070 

846 

546 

708 

768 

1,453 

1,126 

177 

0 

361 

2014  Baseline  Planned 

7,471 

1,127 

619 

583 

1,356 

922 

648 

1,490 

276 

74 

376 

Difference  from  Planned 

-5.6% 

-5.1% 

36.7% 

-6.3% 

-47.8% 

-16.7% 

124.2% 

-24.4% 

-35.9% 

-100.0% 

-4.0% 

Added  Points 

1,756 

119 

236 

329 

1,021 

51 

0 

0 

Test  Point  Growth  Rate 

24.9% 

6.2% 

18.8% 

14.8% 

90.7% 

28.8% 

0% 

0% 

Total  Points  Accomplished  in  201 43 

8,811 

2,035 

1,490 

2,550 

2,147 

228 

0 

361 

Cumulative  SDD  Actual4 

34,888 

11,689 

9,269 

8,322 

3,872 

177 

0 

1,559 

Cumulative  SDD  Planned 

35,683 

11,252 

10,056 

7,399 

4,359 

276 

74 

2,267 

Difference  from  Planned 

-2.2% 

3.9% 

-7.8% 

12.5% 

-11.2% 

-35.9% 

N/A 

-31.2% 

Estimated  Test  Points  Remaining 

22,956 

77 

7,013 

54 

4,049 

150 

4,880 

529 

523 

3,811 

1,870 

1 .  Flight  Sciences  Test  Points  are  shown  separately  for  Block  2B  and  Block  3F.  Flight  envelopes  differ  in  airspeed,  maximum  allowable  g,  and  weapons  carriage,  depending  on  variant. 

2.  Includes  Block  0.5,  Block  1 ,  and  Block  2A  quantities  for  Cumulative  Actual  and  Cumulative  Planned 
3.  Total  Points  Accomplished  =  201 4  Baseline  Accomplished  +  Added  Points 
4.  SDD  -  System  Design  and  Development 
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System 

•  The  F-35  Joint  Strike  Fighter  (JSF)  program  is  a  tri-Service, 
multi-national,  single  seat,  single-engine  family  of  strike 
aircraft  consisting  of  three  variants: 

-  F-35  A  Conventional  Take-Off  and  Landing  (CTOL) 

-  F-35B  Short  Take-Off/Vertical-Landing  (STOVL) 

-  F-35C  Aircraft  Carrier  Variant  (CV) 

•  It  is  designed  to  survive  in  an  advanced  threat  (year  2015  and 
beyond)  environment  using  numerous  advanced  capabilities. 

It  is  also  designed  to  have  improved  lethality  in  this 
environment  compared  to  legacy  multi-role  aircraft. 

•  Using  an  Active  Electronically  Scanned  Array  radar  and  other 
sensors,  the  F-35  is  intended  to  employ  precision-guided 
bombs  such  as  the  Joint  Direct  Attack  Munition  (JDAM)  and 
Joint  Standoff  Weapon,  AIM-120C  radar-guided  Advanced 
Medium-Range  Air-to-Air  Missile,  and  AIM-9  infrared-guided 
short-range  air-to-air  missile. 

•  The  program  provides  mission  capability  in  three  increments: 

-  Block  1  (initial  training,  two  increments  were  fielded: 
Blocks  1A  and  IB) 


-  Block  2  (advanced  training  in  Block  2A  and  limited  combat 
in  Block  2B  ) 

-  Block  3  (limited  combat  in  Block  3i  and  full  combat  in 
Block  3F) 

•  The  F-35  is  under  development  by  a  partnership  of  countries: 
the  United  States,  Great  Britain,  Italy,  the  Netherlands,  Turkey, 
Canada,  Australia,  Denmark,  and  Norway. 

Mission 

•  A  force  equipped  with  F-35  units  should  permit  the  Combatant 
Commander  to  attack  targets  day  or  night,  in  all  weather,  and 
in  highly-defended  areas  of  joint  operations. 

•  F-35  will  be  used  to  attack  fixed  and  mobile  land  targets, 
enemy  surface  units  at-sea,  and  air  threats,  including  advanced 
cruise  missiles. 

Major  Contractor 

Lockheed  Martin,  Aeronautics  Division  -  Fort  Worth,  Texas 


Test  Strategy,  Planning,  and  Resourcing 

•  In  March,  DOT&E  recommended  to  the  USD(AT&L)  that  the 
Block  2B  Operational  Utility  Evaluation  (OUE),  which  was 
being  planned  to  occur  in  mid-2015  in  accordance  with  the 
approved  Test  and  Evaluation  Master  Plan  (TEMP),  should 
not  be  conducted.  Instead,  resources  should  be  focused 
on  conducting  limited  assessments  of  Block  2B  capability 
and  re-allocated  to  assure  the  completion  of  development 
and  testing  of  Block  3i  and  Block  3F  capabilities.  This 
recommendation  was  based  on  DOT&E ’s  review  of  Block  2B 
progress  and  assessment  of  the  program’s  ability  to  start  the 
Block  2B  OUE  as  planned  without  creating  a  significant 
impact  to  Block  3F  development. 

-  The  factors  that  led  to  the  DOT&E  recommendation 
include:  poor  operational  suitability,  an  inability  to 
prepare  pilots  with  adequate  training  and  approved 
tactics  on  the  planned  schedule,  and  the  deferral  to 
Block  3  of  operationally-relevant  deficiencies  that  would 
affect  performance.  It  was  clear  in  March  that  aircraft 
availability  for  operational  testing  would  be  driven 

by  the  long  timelines  required  to  modify  and  retrofit 
the  early  production  operational  test  aircraft  to  the 
Block  2B  configuration,  which  would  not  be  complete 
until  mid-2016.  DOT&E  assessed  that  delaying  the 
Block  2B  OUE  until  late  2016,  as  opposed  to  cancelling 
it,  would  have  a  negative  impact  on  the  program’s  ability 
to  complete  development  of  the  full  Block  3F  combat 
capability  in  a  timely  manner. 

-  In  April,  in  coordination  with  the  Service  Acquisition 
Executives  and  the  JSF  Program  Executive  Officer,  the 
USD(AT&L)  agreed  with  the  DOT&E  recommendation 
and  approved  revising  the  operational  test  period  that 


was  allocated  for  the  Block  2B  OUE  in  the  TEMP  into  a 
re-scoped  effort  of  assessing  the  limited  Block  2B  set  of 
capabilities.  The  JSF  Operational  Test  Team,  JSF  Program 
Office,  and  the  Services’  operational  test  agencies  began 
re-planning  the  Block  2B  operational  test  period  and 
activities. 

-  By  the  middle  of  October,  five  of  the  six  F-35  A  operational 
test  aircraft  assigned  to  the  Edwards  AFB,  California, 
operational  test  squadron  had  been  converted  to  the 
Block  2B  configuration  and  loaded  with  a  version  of 
Block  2B  software  equivalent  to  the  one  being  flown  on  the 
developmental  test  aircraft.  The  sixth  F-35  A  operational 
test  aircraft  began  an  extended  modification  period  at 

the  depot  in  September  and  is  scheduled  to  be  returned 
to  Edwards  AFB  in  February  2015  in  the  Block  2B 
configuration.  These  operational  test  aircraft,  although 
not  in  a  full  Block  2B  operationally-representative 
configuration  as  would  have  been  necessary  to  start  the 
OUE,  will  be  used  to  accomplish  both  developmental  and 
operational  testing  events.  They  will  be  loaded  with  the 
latest  version  of  Block  2B  software  as  it  becomes  available 
and  is  determined  airworthy  for  operational  test  purposes. 

-  Program  schedule  pressures  that  caused  DOT&E  to 
recommend  not  completing  the  Block  2B  OUE  as  planned 
increased  throughout  CY14.  For  example.  Block  2B 
flight  testing,  which  was  scheduled  to  be  complete  in 
October  2014,  is  now  projected  by  the  Program  Office  to 
complete  in  January  2015.  Aircraft  depot  modification 
plans  are  another  example.  The  program  developed 
plans  to  upgrade  fielded  production  aircraft  from  Lots  3 
through  5,  which  includes  operational  test  aircraft  planned 
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for  use  in  the  OUE,  to  the  full  Block  2B  configuration. 
These  plans  show  that  all  of  the  operational  test  aircraft 
which  were  planned  for  the  Block  2B  OUE  will  not  be 
in  the  full  Block  2B  configuration  until  September  2016, 

21  months  later  than  would  have  been  needed  to  conduct 
the  OUE. 

•  DOT&E  conditionally  approved  Revision  4  of  the  TEMP 
in  March  2013,  under  the  provision  that  the  program  revise 
the  master  schedule  so  that  there  was  no  overlap  of  spin-up 
training  for  10T&E  and  the  certification  period  needed  for  the 
Services’  airworthiness  authorities  to  approve  a  flight  clearance 
with  the  software  to  be  used  for  IOT&E.  Specifically,  this 
woidd  require  the  program  to  adjust  the  start  of  the  spin-up 
training  from  February  to  July  2017,  coinciding  with  an 
Operational  Test  Readiness  Review.  This  adjustment  also 
moved  the  start  of  IOT&E  to  January  2018,  vice  August  2017, 
and  hence  pushed  the  completion  of  IOT&E  into  FY19.  In 
spite  of  the  conditional  approval,  the  program  continues  to 
show  schedules  that  plan  for  the  start  of  spin-up  training  in 
February  2017  and  the  start  of  IOT&E  in  August  2017.  In 
addition  to  the  justifications  for  adjusting  the  schedule  that 
DOT&E  outlined  in  the  March  2013  TEMP  conditional 
approval  memo,  the  program  has  encountered  more 
challenges  to  meeting  the  planned  schedule  to  start  IOT&E  in 
August  2017  and  completing  System  Design  and  Development 
(SDD)  in  2018.  These  challenges  include: 

-  Block  3i  flight  testing  began  in  late  May  2014,  five  months 
later  than  the  program’s  baseline  plan. 

-  Block  3F  flight  testing  was  scheduled  to  start  in 
November  2014  according  to  the  program’s  baseline  plan; 
current  program  estimates  show  the  testing  starting  no 
earlier  than  late  February  2015,  three  months  late. 

-  Modification  plans  for  the  IOT&E  aircraft  will  likely  not 
have  aircraft  ready  to  begin  the  start  of  spin-up  training 
in  February  2017  as  planned  by  the  errant  schedule 
submitted  in  the  TEMP.  To  become  Block  3F  capable,  the 
operational  test  aircraft  require  extensive  modifications, 
including  new  processors,  in  addition  to  those  needed  for 
Block  2B  capability.  Block  3F  modification  plans  are 
taking  into  consideration  some  modifications  that  already 
have  engineering  solutions  and  approved  designs.  Other 
modifications  -  although  known  to  be  required  -  are  still 
in  the  formal  change  approval  process  leading  to  parts 
and  modification  kits  being  developed  and  procured  from 
suppliers.  Some  of  these  latter  modifications  are  currently 
not  scheduled  to  be  available  until  May  2017  for  the  F-35A 
and  February  2018  for  the  F-35C,  which  is  later  than 
needed  to  support  spin-up  training  for  IOT&E. 

-  There  is  carryover  of  incomplete  work  from  Block  2B 
development  into  Block  3.  In  coordination  with  the 
Services,  the  program  completed  a  review  in  June  of 
1,151  open  deficiency  reports  identified  during  Block  2B 
development  and  earlier.  Of  these,  572  were  rated  as 
relevant  to  and  affecting  Block  2B  capability;  579  were 
carried  over  for  consideration  for  corrections  in  Block  3. 


-  The  program  removed  test  points  that  were  originally 
planned  to  be  flown  to  support  Block  2B  fleet  release 
(approximately  1,000  mission  systems  test  points);  some 
of  these  points  may  carry  over  and  need  to  be  flown  during 
Block  3F  development. 

-  In  order  to  account  for  these  realities  and  reduce  the 
overlap  of  spin-up  training  for  IOT&E  with  final 
development  activities  (such  as  the  activities  that  provide 
the  certifications  for  use  of  the  final  configuration),  the 
program  master  schedule  should  be  adjusted  to  reflect  these 
realities  and  depict  the  start  of  spin-up  training  for  IOT&E 
no  earlier  than  the  Operational  Test  Readiness  Review  in 
November  2017,  and  the  start  of  IOT&E  for  Block  3F  to 
occur  six  months  later,  in  May  2018  and  completing  in 
May  2019.  If  it  becomes  apparent  that  spin-up  training 
entry  criteria  (e.g.,  providing  properly  configured 
production-representative  aircraft  in  sufficient  numbers) 
cannot  be  met  on  this  timeline,  then  the  schedule  will  have 
to  be  adjusted  again. 

•  This  report  reviews  the  program  by  analyzing  the  progress  of 
testing  and  the  capability  delivered  as  a  function  of  test  results. 
The  program  plans  a  specific  set  of  test  points  (discrete 
measurements  of  performance  under  specific  test  conditions) 
for  accomplishment  in  a  given  calendar  year.  In  this  report, 
test  points  planned  for  a  given  calendar  year  are  referred  to 

as  baseline  test  points.  In  addition  to  baseline  test  points, 
the  program  accomplishes  test  points  added  for  discovery, 
regression  of  new  software,  and  verification  of  fixes  to 
deficiencies  identified  in  flight  test;  these  additional  points  are 
referred  to  as  “growth”  points  in  this  report.  Cumulative  SDD 
test  point  data  refer  to  the  total  progress  towards  completing 
development  at  the  end  of  SDD. 

F-35A  Engine  Failure 

•  An  F-35A  aircraft  assigned  to  the  training  center  at 

Eglin  AFB,  Florida  experienced  an  engine  failure  on  take-off 
on  June  23,  2014.  The  aircraft  was  a  Lot  4  production 
aircraft,  delivered  to  Eglin  AFB  in  June  2013,  and  had  flown 
approximately  160  hours  prior  to  the  incident. 

•  Asa  result  of  the  engine  failure,  the  Program  Office  and 
the  Services  initiated  a  series  of  actions  that  affected  flight 
operations  for  both  the  fielded  production  aircraft  and  the  test 
aircraft. 

-  The  Program  Office  instituted  an  operational  pause  to  flight 
testing  at  the  test  centers  on  June  25,  and  the  contractor 
suspended  acceptance  flight  operations  at  the  production 
plant. 

-  A  fleet-wide  stop  order  was  issued  by  the  Program  Office 
on  July  4,  which  officially  suspended  flight  operations  and 
ground  engine  runs.  This  order  also  initiated  requirements 
to  visually  inspect  the  affected  engine  components  using 
special  equipment  called  a  borescope. 

-  On  July  8,  the  program  began  lifting  restrictions  by 
permitting  engine  runs  up  to  30  percent  power  for  engines 
that  had  completed  the  borescope  inspections. 
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-  On  July  16,  the  program  began  permitting  limited  flight 
operations  for  F-35B  and  F-35C  aircraft  with  stringent 
flight  limitations  and  continued  inspection  requirements. 

-  Aircraft  operating  limitations  have  been  incrementally 
revised  to  permit  flight  testing  to  continue.  By 
mid-September,  the  flight  sciences  aircraft  of  each 
variant  had  been  cleared  to  continue  testing  without 
engine-imposed  envelope  restrictions.  The  rest  of  the 
test  fleet  continues  to  conduct  flight  testing,  but  under 
a  restricted  flight  envelope.  The  program  plans  to 
have  all  engine-imposed  restrictions  removed  from  the 
developmental  test  fleet  by  the  end  of  February  2015,  after 
modifications  to  the  engines  of  each  aircraft  are  complete. 

-  On  October  10,  the  program  confirmed  that  excessive 
rubbing  between  the  hard  polyamide  seal  of  the  second 
stage  stator  and  the  titanium  interface  of  the  integrated 
blade  third  stage  rotor  led  to  the  engine  failure.  This 
excessive  rubbing  occurred  on  a  previous  flight  while 
maneuvering  within  the  limited,  cleared  training  envelope. 
Friction  from  the  rubbing  created  excessively  high 
temperatures  within  the  titanium  rotor,  creating  small 
cracks  that  eventually  led  to  catastrophic  failure  of  the 
rotor  during  the  take-off  on  June  23.  It  is  not  clear  what 
occurred  differently  than  expected  in  the  air  vehicle  and/or 
engine  that  caused  the  excessive  rubbing. 

•  Inspections  of  the  engines  on  all  variants  led  to  discoveries  on 
nine  production  and  test  aircraft  requiring  engine  replacement. 

•  As  of  July  23,  restrictions  on  the  flight  test  aircraft  blocked 
53  percent  (1,357  of  2,575)  of  the  remaining  Block  2B  test 
points;  however,  test  points  have  incrementally  become 
available  as  the  flight  restrictions  were  relaxed  on  some  of 
the  test  aircraft  beginning  in  September  after  the  test  centers 
complied  with  actions  found  necessary  by  the  root  cause 
analysis. 

•  Resolution  of  the  way  forward  with  the  engines  in  test  and 
production  aircraft  was  ongoing  at  the  time  of  this  report. 

-  The  program  developed  and  tested  an  engine  “rub-in” 
procedure.  This  procedure  is  designed  to  ensure  the 
engines  have  sufficient  clearance  between  the  rotors  and 
seals  to  prevent  excessive  rubbing  during  maneuvering. 

The  rub-in  process  is  accomplished  through  two  flights 
during  which  a  specific  profile  is  flown  to  accomplish 
the  procedure,  followed  by  inspections.  As  flight  test 
jets  completed  the  rub-in  procedure,  they  were  cleared  to 
accomplish  some  of  the  blocked  test  points  and  fly  within 
an  expanded,  although  still  limited,  flight  envelope. 

-  The  program  is  developing  an  interim  redesign  of  the  seal, 
which  will  have  grooves  pre-cut  in  the  polyamide  material 
to  provide  clearance  between  the  seal  and  the  rotor  and 
will  prevent  excessive  rubbing  during  maneuvering.  A 
prototype  of  this  “pre-trenched”  seal  was  flight  tested 

in  October  and  is  being  installed  in  the  engines  of  each 
developmental  test  aircraft. 

-  The  program  is  working  with  the  engine  contractor  to 
develop  a  new  redesigned  seal  for  production  engines.  Plans 
on  a  final  design  were  not  complete  at  the  time  of  this  report. 


F-35A  Flight  Sciences 

Flight  Test  Activity  with  AF- 1,  AF-2,  and  AF-4  Test  Aircraft 

•  F-35A  flight  sciences  testing  focused  on: 

-  Completing  the  full  Block  2B  flight  envelope 

-  High  angle  of  attack  testing  (clean  wing  for  Block  2B 
and  with  external  stores  for  Block  3) 

-  Ground  and  flight  testing  of  the  redesigned  fuel  tank 
ullage  inerting  system  (i.e.,  inerting  of  the  space  not 
occupied  by  fuel  in  a  fuel  tank),  consisting  of  the 
On-Board  Inert  Gas  Generation  System  (OBIGGS)  and 
associated  fuel  pressurization  and  ventilation  system 

-  Start  of  Block  3F  loads  and  flying  qualities  testing, 
predominantly  flying  with  externally-loaded 
air-to-ground  and  air-to-air  weapons 

-  Regression  testing  of  updated  versions  of  vehicle 
systems  software 

-  Testing  of  the  aerodynamic  loads  in  the  gun  bay. 

(Note:  Block  3F  F-35A  aircraft  will  have  an  internal 
gun;  F-35B  and  F-35C  aircraft  will  use  a  podded  gun 
mounted  on  the  center  fuselage  station.) 

•  Restrictions  imposed  on  the  fleet  from  the  June  engine 
failure  coupled  with  the  focus  on  Block  2B  mission 
systems  testing  hampered  progress  in  F-35A  flight  sciences 
testing. 

•  Excessive  free-play  in  the  rudder  hinges  on  AF-2  required 
extended  downtime  for  repair.  These  repairs  occurred  in 
July  during  the  period  of  restrictions  from  the  engine  fire. 

F-35A  Flight  Sciences  Assessment 

•  Through  the  end  of  November,  test  point  accomplishment 
in  CY14  was  6  percent  behind  the  plan  for  accomplishing 
Block  2B  points  and  48  percent  behind  for  Block  3F.  The 
test  team  flew  25  percent  fewer  test  flights  than  planned  for 
the  year  (197  flown;  262  planned).  Prioritization  of  flight 
test  resources  to  focus  on  mission  systems  flight  testing  for 
Block  2B  at  the  Edwards  AFB  test  center  (where  mission 
systems  and  F-35A  flight  sciences  testing  are  conducted) 
reduced  the  opportunity  for  flight  science  testing  to  achieve 
planned  progress  in  Block  3F  testing. 

•  The  plan  for  Block  2B  test  points  was  adjusted  in  CY14, 
resulting  in  the  net  reduction  of  343  of  926  (37  percent) 
of  the  original  points  planned  for  the  year.  The  program 
designated  these  points  as  no  longer  required  for  Block  2B 
fleet  release. 

•  Restrictions  imposed  from  the  J  une  engine  failure  initially 
blocked  access  to  almost  all  (254  of  261)  remaining 
Block  2B  flight  sciences  test  points.  The  program  was 
able  to  relax  the  restrictions  on  an  aircraft-by-aircraft 
basis  beginning  in  September,  providing  access  to  some 
of  the  blocked  test  points;  all  points  were  available  as  of 
the  end  of  October.  The  prioritization  of  mission  systems 
testing  coupled  with  the  restrictions  from  the  engine  failure 
created  a  debt  of  flight  sciences  testing  on  the  F-35A  that 
will  need  to  be  overcome  in  CY15  and  early  CY16  for 

the  program  to  maintain  Block  3F  flight  envelope  release 
schedule. 
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•  The  program  added  236  flight  sciences  test  points 
through  the  end  of  November,  equating  to  a  growth  rate 
of  19  percent,  which  is  near  the  planned  growth  rate  of 
17  percent. 

•  AF-4  underwent  a  modification  from  March  through  May, 
during  which  the  redesigned  fuel  tank  ullage  inerting 
system  was  installed.  This  modification  and  testing  is  part 
of  the  effort  to  address  deficiencies  in  lightning  protection 
and  vulnerability  reduction  to  ballistic  threats.  Testing  to 
assess  on-the-ground  inerting  performance  of  the  redesign 
and  to  validate  modeling  results  was  completed  in  May. 
Flight  testing  to  assess  the  fuel  system  pressurization  and 
ventilation  capability  of  the  redesign  was  mostly  completed 
in  June;  dive  test  points  were  blocked  by  restrictions 
imposed  by  the  engine  failure.  Further  testing  to  assess 
corrections  to  the  redesign  is  scheduled  to  occur  in 
December  2014. 

•  Discoveries  in  F-35A  flight  sciences  testing: 

-  Higher  than  expected  wear  in  the  rudder  hinges  of  AF-2 
was  discovered  during  routine  inspections,  following 
flight  testing  in  regions  of  the  envelope  where  higher 
dynamic  loads  are  exerted  on  the  rudder  surfaces. 
Replacement  of  the  clevis  of  the  middle  rudder  hinges 
was  necessary,  and  additional  inspections  to  check  rudder 
free  play  are  required. 

-  AF-4  encountered  a  blown  tire  and  damage  to  the  main 
landing  gear  while  conducting  crosswind  landing  testing 
in  February,  requiring  a  two-week  down  period  for 
repairs. 

-  Ground  testing  on  aircraft  AF-4  revealed  that  pressure 
from  the  OBIGGS  inadvertently  pushes  fuel  between 
tanks.  Per  engineering  directive,  the  test  team  removed 
and  capped  the  inert  air  distribution  lines  that  were 
causing  the  fuel  transfer  as  a  temporary  measure  to 
permit  AF-4  to  continue  developmental  testing  of  other 
(non-OBIGGS)  test  requirements.  Further  modifications 
to  software  and  the  addition  of  a  control  valve  were 
made  to  AF-4  in  November  for  testing  planned  for 
December  2014. 

-  Inerting  the  aircraft  on  the  ground  with  external  nitrogen 
forces  fuel  to  vent  from  the  fuel  tanks  under  certain  fuel 
states.  The  procedure  to  purge  the  fuel  system  with 
external  nitrogen  was  introduced  with  the  redesigned 
ullage  inerting  system  to  provide  lightning  protection 
on  the  ground.  The  program  plans  to  address  this  fuel 
venting  by  testing  two  additional  check  valves  on  AF-4 
for  incorporation  into  the  final  design. 

•  Weight  management  of  the  F-35A  is  important  for  meeting 
air  vehicle  performance  requirements  and  structural  life 
expectations.  These  estimates  are  based  on  measured 
weights  of  components  and  subassemblies,  calculated 
weights  from  approved  design  drawings  released  for 
build,  and  estimated  weights  of  remaining  components. 
These  estimates  are  used  to  predict  the  weight  of  the  first 
Lot  7  F-35A  aircraft  (AF-72),  planned  for  delivery  in 


August  2015,  which  will  be  the  basis  for  evaluating  contract 
specification  compliance  for  aircraft  weight. 

-  According  to  these  reports,  the  program  has  reduced 
weight  by  16  pounds  in  CY 14  (from  January  to  October 
estimate).  The  current  estimate  of  29,016  pounds  is 

355  pounds  (1.2  percent)  below  the  planned  not-to-exceed 
weight  of  29,371  pounds. 

-  The  program  has  demonstrated  positive  weight 
management  of  the  F-35A  over  the  past  38  months, 
showing  a  net  loss  of  123  pounds  in  the  estimates  from 
August  2011  to  October  2014.  The  program  will  need  to 
ensure  the  actual  aircraft  weight  meets  predictions,  as  well 
as  continue  rigorous  management  of  the  actual  aircraft 
weight  beyond  the  technical  performance  measurements 
of  contract  specification  in  CY15  through  the  balance  of 
SDD  to  avoid  perfonnance  degradation  that  would  affect 
operational  capability. 

F-35B  Flight  Sciences 

Flight  Test  Activity  with  BF- 1,  BF-2,  BF-3,  BF-4,  and  BF-5  Test 

Aircraft 

•  F-35B  flight  sciences  focused  on: 

-  Continued  expansion  of  the  Block  2B  flight  envelope, 
including  weapons  separation  testing 

-  High  angle  of  attack  testing 

-  Wet  runway  testing  (completed  with  BF-4  in  May  at 
Edwards  AFB) 

-  Testing  of  landing  control  authority  in  crosswind 
conditions 

-  Testing  with  external  air-to-air  and  air-to-ground  weapons 
(Block  3F  capability) 

-  STOVL  mode  flight  operations 

-  Testing  of  fuel  dump  capability  with  a  new  valve  and  seals 

-  Ground  and  flight  testing  of  the  redesigned  ullage  inerting 
system 

-  Flight  testing  in  support  of  expeditionary  operations 
(i.e.,  landing  on  matted  runways,  AM-2  padding) 

-  Preparations  for  and  conducting  climatic  testing  on  BF-5 
in  the  climatic  chamber 

F-35B  Flight  Sciences  Assessment 

•  Through  the  end  of  November,  test  point  accomplishment 
for  CY14  was  5  percent  behind  the  plan  for  accomplishing 
Block  2B  points  and  37  percent  ahead  of  the  plan  for 
Block  3F  points.  Test  flights  were  slightly  ahead  of  plan 
(313  flown;  296  planned).  The  test  force  maintained  test 
point  productivity  by  accomplishing  test  points  from  the 
Block  3F  test  plan  for  flying  qualities,  air  data,  propulsion, 
and  loads  in  the  STOVL  mode  and  with  external  stores.  The 
program  projects  the  completion  of  Block  2B  flight  sciences 
testing  to  occur  by  the  end  of  December  2014,  two  months 
later  than  planned. 

•  This  projection  follows  adjustments  made  by  the  Program 
Office  to  the  plan  for  Block  2B  test  points  in  CY14,  which 
resulted  in  the  net  reduction  of  394  out  of  1,545  (26  percent) 
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of  the  points  planned  for  the  year.  These  points  were 
reviewed  by  the  contractor  and  the  Program  Office,  and 
designated  as  no  longer  required  for  Block  2B  fleet  release 
and  Marine  Corps  IOC.  This  reduction  brought  the  total 
2014  plan  to  1,151  points,  1,127  of  which  were  planned  to  be 
completed  by  the  end  of  November. 

•  Crosswind  landing  testing  in  the  conventional  landing  mode 
(not  vertical  landing)  was  not  completed;  but  sufficient 
testing  was  accomplished  to  clear  landings  up  to  20  knots 
of  crosswind,  short  of  the  ORD  requirement  of  25  knots  of 
crosswind. 

•  BF-4  was  modified  with  the  redesigned  fuel  tank  inerting 
system  late  in  CY13.  Testing  to  assess  ground  inerting 
performance  and  validate  results  from  the  fuel  system 
simulator  -  a  full  mock-up  surrogate  of  the  F-35B  fuel 
system  -  was  completed  in  December  2013.  Further 
testing  of  the  tank  inerting  system  did  not  occur  until 
September  2014,  as  other  test  requirements  (i.e.,  wet 
runway  testing)  needed  to  be  conducted  with  BF-4,  and 
known  deficiencies  needed  to  be  addressed  with  corrections 
to  software.  Flight  testing  of  the  tank  inerting  system 

is  ongoing.  Regression  testing  to  verify  correction  of 
deficiencies  in  the  redesign  discovered  from  ground  testing 
(on  the  aircraft  and  in  the  simulator)  was  conducted  in 
early  October  and  will  continue  in  December  after  updated 
software  is  released  to  the  test  aircraft  for  flight  testing. 

•  Discoveries  in  F-35B  flight  sciences  testing  included: 

-  Early  fuel  dump  testing  in  2011  discovered  that  fuel  does 
not  completely  eject  overboard,  but  collects  in  the  area 
between  the  flaperons  and  the  aircraft  structure  and  runs 
inboard  toward  the  Integrated  Power  Package  exhaust 
outlet,  creating  a  potential  fire  hazard.  Testing  of  a 
redesigned  dump  nozzle,  improved  seals  for  the  flaperons, 
and  heat-shrinkable  tubing  added  to  wiring  harnesses  for 
protection  in  the  event  of  fuel  wetting  have  all  contributed 
to  a  new  fuel  dumping  procedure. 

-  Inerting  performance  in  certain  fuel  tanks  during  ground 
testing  of  the  redesigned  ullage  inerting  system  did 

not  meet  the  performance  demonstrated  during  fuel 
system  simulator  testing.  To  address  this  discrepancy, 
an  additional  OBIGGS  distribution  line  was  installed  on 
aircraft  BF-4.  The  discovery  affects  all  variants;  retrofit 
kits  have  been  developed  for  the  F-35A  and  F-35C 
variants. 

-  The  redesigned  ullage  inerting  system  has  the  potential 
to  generate  pressure  spikes  when  pressure  in  the  aerial 
refueling  manifold  is  released  into  the  fuel  tanks.  A 
blanking  plate  was  installed  on  BF-4  to  isolate  the  aerial 
refueling  manifold  from  the  OBIGGS  as  a  temporary 
measure  to  allow  it  to  ferry  to  Edwards  AFB  to  conduct 
testing  on  wet  runways.  A  software  modification  of  the 
valve  control  logic  was  tested  in  late  September,  allowing 
removal  of  the  blanking  plate. 

-  The  aircraft  does  not  maintain  residual  inerting  after  flight 
for  the  required  interval  of  12  hours,  which  is  a  lightning 


protection  requirement.  Residual  inerting  is  a  result  of 
the  inert  air  produced  by  the  OBIGGS  remaining  in  the 
ullage  area  of  the  fuel  tanks  after  a  flight.  The  program  is 
investigating  a  correction  to  this  problem.  If  the  residual 
inerting  cannot  be  improved,  aircraft  maintainers  will  be 
required  to  purge  fuel  tanks  with  external  nitrogen  more 
frequently  or  alternative  lightning  protection  strategies 
(e.g.,  lightning-protected  shelters,  will  have  to  be  adopted. 

-  In  heavy  buffet  conditions,  which  occur  between  20  and 
26  degrees  angle  of  attack,  faults  occurred  in  the  inertial 
measurement  units  (IMUs)  in  the  aircraft  that  degraded 
the  flight  control  system  (two  of  three  flight  control 
channels  become  disabled),  requiring  a  flight  abort.  This 
condition  blocked  28  test  points  needed  for  the  Block  2B 
fleet  release.  The  program  made  adjustments  to  the  flight 
control  software,  which  were  tested  in  late  October  and 
the  test  points  were  unblocked,  enabling  some  testing  in 
the  heavy  buffet  conditions  to  continue.  However,  nine 
additional  test  points  needed  for  the  Block  2B  fleet  release 
remained  blocked  at  the  end  of  November  because  of  high 
dynamic  loads  on  the  rudder  at  lower  altitudes,  in  the  same 
angle  of  attack  range,  and  require  additional  analyses  and 
mitigation  to  complete. 

•  Weight  management  of  the  F-35B  aircraft  is  critical  to 
meeting  the  Key  Performance  Parameters  (KPPs)  in  the 
ORD,  including  the  vertical  lift  bring-back  requirement. 

This  KPP  requires  the  F-35B  to  be  able  to  fly  an 
operationally  representative  profile  and  recover  to  the  ship 
with  the  necessary  fuel  and  balance  of  unexpended  weapons 
(two  1,000-pound  bombs  and  two  AIM-120  missiles)  to 
safely  conduct  a  vertical  landing.  These  estimates  are  based 
on  measured  weights  of  components  and  subassemblies, 
calculated  weights  from  approved  design  drawings  released 
for  build,  and  estimated  weights  of  remaining  components. 
These  estimates  are  used  to  predict  the  weight  of  the  first 
Lot  7  F-35B  aircraft  (BF-44),  planned  for  delivery  in 
August  2015,  which  will  be  the  basis  for  evaluating  contract 
specification  compliance  for  aircraft  weight. 

-  Weight  reports  for  the  F-35B  as  of  October  show  that  the 
program  added  18  pounds  to  the  estimated  weight  in  CY14 
and  a  net  addition  of  82  pounds  over  the  last  38  months 
(August  2011  to  October  2014).  The  current  estimate 

of  32,412  pounds  is  337  pounds  (1  percent)  below  the 
objective  vertical  lift  bring-back  not-to-exceed  weight  of 
32,749  pounds. 

-  Managing  weight  growth  for  the  F-35B  will  continue 
to  be  a  challenge  in  light  of  the  small  weight  margin 
available  and  the  possibility  for  continued  discovery 
through  the  remaining  SDD  phase,  which  extends  two 
years  past  the  delivery  of  the  first  Lot  7  aircraft,  planned 
for  August  2015.  The  program  will  need  to  ensure  actual 
weights  meet  predictions.  Known  modifications  and 
retrofits  for  production  aircraft  in  Lots  2  through  6  will 
add  weight  to  those  aircraft,  varying  from  210  pounds  for 
the  Lot  3  aircraft  to  17  pounds  for  the  Lot  6  aircraft.  In 
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addition,  the  program  is  currently  redesigning  the  FS496 
bulkhead  for  Lot  9  production  aircraft  and  later  as  a  result 
of  the  failure  of  that  bulkhead  in  the  ground  test  article 
during  durability  testing.  The  effect  of  the  redesigned 
bulkhead  on  the  weight  of  the  aircraft  is  not  yet  known. 


•  The  following  table,  first  displayed  in  the  FY 1 1  Annual 
Report  and  updated  each  year,  describes  observed  door 
and  propulsion  problems  by  component  and  identifies  the 
production  cut-in  of  the  correction  or  update,  if  known. 


F-35B  DOOR  AND  PROPULSION  PROBLEMS 

Category 

Component 

Problem 

Design  Fix  and  Test  Status 

Production  Cut-In 

Structure 

Auxiliary  Air  Inlet 
Door(AAID) 

Inadequate  life  on  door  locks,  excessive 
wear  and  fatigue  due  to  the  buffet 
environment,  inadequate  seal  design. 

New  designed  doors  are  being  installed  on  Low-Rate 

Initial  Production  (LRIP)  aircraft  as  part  of  the  on-going 
modification  plan;  1 4  completed  through  the  end  of 
September.  Fatigue  testing  of  the  doors  started  in 

November  201 2  and  completed  the  planned  2  lifetimes  of 
testing  at  the  end  of  September  2014.  Inspections  were 
ongoing  as  of  the  end  of  November,  with  no  discoveries.  Fix 
appears  to  resolve  problem. 

BF-38 

LRIP  Lot  6 

2014 

Propulsion 

Drive  Shaft 

Lift  fan  drive  shaft  is  undergoing  a  second 
redesign.  Original  design  was  inadequate 
due  to  shaft  stretch  requirements  to 
accommodate  thermal  growth,  tolerances, 
and  maneuver  deflections.  First  redesign 
failed  qualification  testing. 

New  design  completed  qualification  testing  and  appears 
to  reduce  the  problem.  Full  envelope  requirements 
are  currently  being  met  on  production  aircraft  with  an 
interim  design  solution  using  spacers  to  lengthen  the 
early  production  drive  shaft.  New  design  is  dependent  on 
updated  propulsion  software  load  planned  to  be  available 
by  Lot  9. 

BF-56 

LRIP  Lot  9 

2016 

Propulsion 

Clutch 

Lift  fan  clutch  has  experienced  higher 
than  expected  drag  heating  during 
conventional  (up  and  away)  flight  during 
early  testing. 

New  clutch  plate  design,  with  more  heat-tolerant  material, 
is  complete.  Clutch  plates  are  being  thinned  on  Lot  5  and  6 
aircraft,  at  the  expense  of  reduced  life  (engagements)  to 
the  clutch,  to  prevent  drag  heating.  Solutions  appear  to  be 
effective;  very  few  hot  clutches  are  experienced  in  fleet  wide 
operations  now. 

TailTBD 

Mid-LRIPLot8 

2015 

Propulsion 

Roll  Post  Nozzle 
Actuator 

Roll  post  nozzle  bay  temperatures  exceed 
current  actuator  capability;  insulation 
is  needed  to  prevent  possible  actuator 
failure  during  vertical  lift  operations. 

Insulation  between  the  roll  post  nozzle  bay  and  the  actuators 
is  being  installed  in  pre-Lot  6  aircraft  to  allow  unrestricted 
operations,  however  the  actuators  must  be  replaced  at 

1,000  hour  intervals.  New  actuators  will  be  installed  in  Lot 

6  aircraft  and  beyond,  removing  the  requirements  for  the 
insulation  and  extending  the  service  life  to  4,000  hours. 

BF-38 

LRIP  Lot  6 

2015 

Propulsion 

Lift  Fan 

Inter  Stage  Vanes 
(ISV) 

Vanes  between  stages  of  the  lift  fan 
experience  excessive  vibration/flutter 
during  mode  4  flight  when  temperature 
is  below  5°F  or  above  1 07°F  degrees  and 
speed  is  greater  than  130  knots  calibrated 
airspeed. 

Aircraft  are  restricted  from  mode  4  flight  outside  the 
temperature  and  speed  restrictions  noted.  A  unit  level  Time 
Compliant  Technical  Directive  is  being  accomplished  for  48 
fielded  lift  fans  to  replace  the  ISVs  with  a  new  ISV  made  of 
more  durable  material  tolerant  over  a  greater  temperature 
range,  with  production  cut  in  on  new  Lift  Fans. 

New  vanes 
retrograded  in 
fielded  aircraft, 
incorporated  in 
new  production 
lift  fans 

F-35C  Flight  Sciences 

Flight  Test  Activity  with  CF- 1,  CF-2,  CF-3,  and  CF-5  Test  Aircraft 

•  F-35C  flight  sciences  focused  on: 

-  Structural  survey  testing  of  the  newly  designed  arresting 
hook  system  (This  testing  was  a  pre-requisite  for  the 
first  developmental  testing  period  aboard  an  aircraft 
carrier,  referred  to  as  DT-1,  which  was  conducted  in 
November  2014.) 

-  Block  2B  weapons  envelope  and  loads  testing 

-  Block  2B  high  angle  of  attack  testing 

-  Testing  with  external  air-to-air  and  air-to-ground  weapons 
(Block  3F  capability) 

-  Fuel  dump  testing 

•  The  program  modified  CF-3  and  CF-5  with  the  new  arresting 
hook  system  and  modified  nose  landing  gear,  which  was 
necessary  to  prepare  for  and  accomplish  the  first  set  of  ship 
trials,  completed  in  November. 


F-35C  Flight  Sciences  Assessment 

•  Through  the  end  of  November,  test  point  accomplishment  for 
CY14  was  17  percent  behind  the  plan  for  Block  2B  points 
and  124  percent  ahead  for  Block  3F  points.  Test  flights 
were  10  percent  ahead  of  the  plan  (286  flown;  261  planned). 
Similar  to  the  F-35B,  the  test  force  has  been  able  to 
maintain  test  point  productivity  by  completing  points  from 
the  Block  3F  test  plan,  such  as  performance  assessments 
with  external  weapons,  which  were  completed  earlier  than 
planned. 

•  Similar  to  the  other  variants,  the  program  adjusted  the 
plan  for  Block  2B  test  points,  resulting  in  a  net  reduction 
of  81  of  1,003  test  points  (8  percent)  planned  for  the  year. 
These  points  were  designated  as  no  longer  required  for  Block 
2B  fleet  release. 

•  Transonic  Roll-Off  (TRO)  and  airframe  buffet  continue 
to  be  a  program  concern.  All  three  variants  required 
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modifications  of  the  control  laws  to  control  the  effects  of 
transonic  flight  and  buffet  producing  maneuvering.  In 
anticipation  of  difficulty  in  these  flight  regimes,  the  ability 
to  incorporate  spoilers  in  F-35C  aircraft  was  provided  early 
in  the  program.  F-35C  handling  characteristics  in  transonic 
and  buffet-producing  regimes  were  in  need  of  correction  and 
worse  than  in  other  variants.  Flight  testing  with  the  addition 
of  spoilers  is  planned,  but  not  yet  started. 

•  CF-8  (a  mission  systems  test  aircraft  assigned  to  the 
Edwards  AFB  test  force)  was  scheduled  to  undergo 
modifications  to  include  the  redesigned  fuel  tank  inerting 
system  in  June  2014;  however,  the  modification  was  delayed 
pending  conversion  of  CF-8  to  the  Block  3i  configuration. 
The  program  has  scheduled  the  modifications  for 
February  2015,  with  ground  and  flight  testing  to  follow  soon 
after. 

•  Discoveries  included: 

-  The  test  force  flew  test  missions  with  CF-2  in 
December  2013  and  January  2014  to  assess  and 
characterize  the  effects  of  buffet  and  TRO  on  the 
helmet-mounted  displays  and  handling  qualities  while 
conducting  tasks  associated  with  operational  maneuvering 
(basic  offensive  and  defensive  fighter  maneuvers).  Buffet 
affected  display  symbology,  and  would  have  the  greatest 
impact  in  scenarios  where  a  pilot  was  maneuvering  to 
defeat  a  missile  shot. 

-  Deficiencies  in  the  nosewheel  steering  motor  and  the  pitch 
pivot  pin  of  the  arresting  hook  system  slowed  testing  (see 
ship  integration  section  for  details  of  the  arresting  hook 
system  testing). 

•  Weight  management  is  important  for  meeting  air  vehicle 
performance  requirements,  including  the  KPP  for  recovery 
approach  speed  to  the  aircraft  carrier,  and  structural  life 
expectations.  These  estimates  are  based  on  measured 
weights  of  components  and  subassemblies,  calculated 
weights  from  approved  design  drawings  released  for  build, 
and  estimated  weights  of  remaining  components.  These 
estimates  are  used  to  project  the  weight  of  the  first  Lot  8 
F-35C  aircraft  (CF-28),  planned  for  delivery  in  April  2016, 
which  will  be  the  basis  for  evaluating  contract  specification 
compliance  for  aircraft  weight. 

-  The  weight  reports  show  that  the  program  has  reduced 
weight  by  62  pounds  in  CY14  (from  January  to  October 
estimate).  The  current  estimate  of  34,519  pounds  is 
349  pounds  (1  percent)  below  the  planned  not-to-exceed 
weight. 

-  The  program  has  demonstrated  positive  weight 
management  of  the  F-35C  over  the  past  38  months, 
showing  a  net  loss  of  1 03  pounds  in  the  estimates  from 
August  2011  to  October  2014.  The  program  will  need 
to  ensure  the  actual  aircraft  weight  meets  predictions 
and  continue  rigorous  management  of  the  actual  aircraft 
weight  beyond  the  technical  performance  measurements 
of  contract  specification  in  CY16  through  the  balance  of 
SDD  to  avoid  performance  degradation  that  would  affect 
operational  capability. 


Mission  Systems 

Flight  Test  Activity  with  AF-3,  AF-6,  AF-7,  BF-4,  BF-5,  BF- 1 7,  BF-  i  8, 
CF-3,  and  CF-8  Flight  Test  Aircraft  and  Software  Development 
Progress 

•  Mission  systems  are  developed,  tested,  and  fielded  in 

incremental  blocks  of  capability. 

-  Block  1.  The  program  designated  Block  1  for  initial 
training  capability  and  allocated  two  increments:  Block  1A 
for  Lot  2  (12  aircraft)  and  Block  IB  for  Lot  3  aircraft 

(17  aircraft).  No  combat  capability  is  available  in  either 
Block  1  increment.  All  Lot  2  aircraft  have  been  converted 
to  Block  IB;  the  U.S.  Services  currently  have  26  Block  IB 
aircraft  (13  F-35A  in  the  Air  Force  and  13  F-35B  in  the 
Marine  Corps).  Additionally,  two  F-35B  Block  IB  aircraft 
have  been  accepted  by  the  United  Kingdom  and  one  F-35A 
Block  IB  aircraft  by  the  Netherlands;  these  aircraft  are 
currently  assigned  to  the  training  center  at  Eglin  AFB. 

-  Block  2A.  The  program  designated  Block  2A  for  advanced 
training  capability  and  delivered  aircraft  in  production 
Lots  4  and  5  in  this  configuration.  No  combat  capability  is 
available  in  Block  2A.  The  U.S.  Services  have  62  aircraft 
in  the  Block  2A  configuration  (32  F-35A  in  the  Air  Force, 

19  F-35B  in  the  Marine  Corps,  and  1 1  F-35C  in  the  Navy). 
Additionally,  one  F-35B  and  one  F-35A  have  been  accepted 
by  the  United  Kingdom  and  the  Netherlands,  respectively; 
both  aircraft  are  assigned  to  the  training  center. 

-  Block  2B.  The  program  designated  Block  2B  for  initial, 
limited  combat  capability  for  selected  internal  weapons 
(AIM-120C,  GBU-32/31,  and  GBU-12).  This  block 

is  not  associated  with  the  delivery  of  any  production 
aircraft.  Block  2B  software  has  been  in  flight  test  since 
February  2013.  Once  complete  with  flight  test  and 
certification,  Block  2B  software  may  be  retrofitted  onto 
aircraft  from  production  Lots  2  through  5,  provided  the 
necessary  hardware  modifications  have  been  completed 
as  well.  Block  2B  is  planned  to  be  the  Marine  Corps  IOC 
configuration. 

-  Block  3i.  The  program  designated  Block  3i  for  delivery 
of  aircraft  in  production  Lots  6  through  8,  as  these  aircraft 
will  be  built  with  a  set  of  upgraded  integrated  core 
processors  (referred  to  as  Technical  Refresh  2,  or  TR2). 

The  capabilities  associated  with  Block  3i  software  will  vary 
based  on  the  production  lot.  Lot  6  aircraft  are  expected  to 
be  delivered  with  capabilities  equivalent  to  Block  2A  in 
Lot  5,  aircraft  in  Lots  7  and  8  are  planned  to  be  delivered 
with  capabilities  equivalent  to  Block  2B.  Block  3i  software 
began  flight  testing  in  May  2014.  The  program  delivered 
the  first  Block  3i  aircraft,  an  F-35A,  to  Luke  AFB,  Arizona, 
in  late  October.  Four  more  F-35A  aircraft  were  delivered 
to  Luke  AFB  and  one  F-35B  to  Marine  Corps  Air  Station 
(MCAS)  Beaufort,  South  Carolina,  by  the  end  of  November. 

-  Block  3F.  The  program  designated  Block  3F  as  the  full 
SDD  capability  for  production  Lot  9  and  later.  Although 
under  development,  flight  testing  with  Block  3F  software 
on  the  F-35  test  aircraft  has  not  started.  The  program 
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plans  to  begin  flight  testing  in  early  CY15.  Aircraft  from 
production  Lots  2  through  5  will  need  to  be  modified, 
including  the  installation  of  TR2  processors,  to  have 
Block  3F  capabilities. 

Mission  systems  testing  focused  on: 

-  Completing  flight  testing  of  Block  2B  capabilities 

-  Start  of  flight  testing  of  Block  3i  software,  which  began  in 
May 

-  Start  of  Generation  III  helmet-mounted  display  system 
(HMDS)  testing 

-  Multi-ship  data  link  performance  (via  the  multi-platform 
advanced  data  link  (MADL)  system  and  Link  16) 

-  Radar  performance 

-  Troubleshooting  navigation  solution  problems,  which 
caused  a  pause  in  weapon  testing  in  August 

-  Manual  Ground  Collision  Avoidance  System  testing, 
which  was  added  by  the  program  in  CY14  as  a  Block  2B 
capability  to  be  delivered  with  fleet  release 

-  Flight  testing  six  increments  of  Block  2B  software  and  two 
increments  of  Block  3i  software  (note:  the  program  plans 
to  release  another  version  of  3i  software  to  flight  test  prior 
to  the  end  ofCY14) 

-  Block  3F  software  -  first  version  began  testing  on  the 
Cooperative  Avionics  Test  Bed  (first  flight  was  on  J uly  31) 

The  six  mission  systems  flight  test  aircraft  assigned  to 
the  Edwards  AFB  test  center  flew  an  average  rate  of  7.0 
flights  per  aircraft  per  month  in  CY14  through  November, 
exceeding  the  planned  rate  of  5.4  by  30  percent,  and  flew 
121  percent  of  the  planned  flights  (472  sorties  accomplished 
compared  to  390  planned). 

The  program  prioritized  flight  test  activity  to  attempt  to 
complete  Block  2B  flight  testing  by  the  end  of  October  2014, 
per  the  approved  baseline  schedule.  However,  as  of  the 
end  of  November,  87  percent  of  the  total  Block  2B  mission 
systems  baseline  test  points  were  accomplished  (3,654  of 
4,183  total  points  accomplished,  529  points  remaining). 

The  test  team  accomplished  74  percent  of  the  planned 
2014  baseline  mission  systems  test  points  from  test  plans 
for  Blocks  2B  and  3i  by  the  end  of  November  (1,303 
baseline  test  points  accomplished,  1,766  planned).  The 
team  also  accomplished  an  additional  1,072  growth  test 
points.  These  points  were  needed  for  regression  testing 
of  new  revisions  of  Block  2B  software,  identifying  and 
characterizing  deficiencies  in  mission  systems  performance, 
verification  of  corrections  of  deficiencies,  and  other  testing 
the  program  found  necessary  to  add  to  the  baseline  test 
plans.  Although  the  program  plans  for  some  growth  points 
during  development,  the  rate  of  growth  experienced  for 
CY14  through  the  end  of  November  for  Block  2B  testing 
(91  percent)  was  higher  than  the  planned  rate  of  45  percent 
used  by  the  program  for  CY14.  The  growth  rate  for  the 
limited  amount  of  Block  3i  testing  was  29  percent. 

Five  F-35A  operational  test  aircraft  (all  of  which  include 
flight  test  instrumentation  and  recording  equipment  identical 
to  SDD  mission  systems  test  aircraft)  were  modified  and 
loaded  with  a  developmental  test  version  of  Block  2B 


software  -  one  aircraft  in  July,  two  in  August,  one  in 
September,  and  one  in  October.  As  a  result  of  the  decision 
to  not  conduct  the  Block  2B  OUE,  the  program  is  able  to 
use  these  aircraft  to  support  the  effort  to  complete  Block  2B 
developmental  testing.  Depending  on  the  availability 
of  these  aircraft  after  the  Block  3F  modifications  plan 
is  finalized,  they  will  be  available  to  support  re-scoped 
Block  2B  operational  test  activity. 

Mission  Systems  Assessment 

•  Block  2B 

-  Although  test  flight  sortie  goals  were  exceeded,  and 
over  75  percent  of  planned  baseline  test  points  were 
accomplished  as  of  the  end  of  November,  delivery  of 
Block  2B  capability,  and  thus  the  ability  to  complete 
development  by  October,  was  hampered  by  several  factors: 

■  The  need  to  develop,  release,  and  test  unplanned 
versions  of  Block  2B  software  to  improve  stability  and 
fix  deficiencies. 

■  Discoveries  continued  to  occur  in  later  versions  of 
software. 

■  Restrictions  to  flight  test  aircraft  apart  from  those 
imposed  due  to  the  June  engine  failure  reduced  the 
accessible  test  points. 

»  For  example,  flight  operations  with  AF-6  and 
AF-7  mission  systems  test  aircraft  were  suspended 
temporarily  on  June  20  when  the  program  issued 
a  stop  order  on  F-3  5  A  production  aircraft  until 
inspections  were  completed  on  the  nacelle  vent 
inlet  tube.  A  crack  in  the  tube  was  discovered  on  a 
production  F-35A  aircraft  at  Eglin  AFB  following 
an  incident  where  ground  crews  observed  fuel 
leaking  from  the  tube  during  hot  pit  ground  refueling 
operations  on  June  11  (AF-6  and  AF-7  are  Lot  1 
production  aircraft  assigned  to  the  Edwards  AFB  test 
center). 

»  Following  the  inspections,  the  program  released  an 
interim  aircraft  operating  limitation  restricting  F-3  5 A 
production  aircraft  to  3  g’s  and  no  air  refueling.  This 
affects  all  fielded  production  aircraft  as  well,  which 
carry  these  restrictions  concurrent  with  the  restrictions 
related  to  engine  failure,  until  they  are  modified. 

These  restrictions  remained  in  place  on  AF-6  and 
AF-7  until  the  test  center  replaced  the  tubes. 

-  To  date,  performance  of  2BS5  software,  which  began  flight 
testing  in  J  une,  has  shown  improvement  in  startup  and 
inflight  stability  compared  to  earlier  versions.  However, 
fusion  of  information  from  own-ship  sensors,  as  well 

as  fusion  of  information  from  off-board  sensors  is  still 
deficient.  The  Distributed  Aperture  System  continues 
to  exhibit  high  false-alarm  rates  and  false  target  tracks, 
and  poor  stability  performance,  even  in  later  versions  of 
software. 

-  In  June,  the  Program  Office  and  the  Services  completed 
a  review  of  nearly  1,500  deficiency  reports  accumulated 
since  the  beginning  of  testing  to  adjudicate  the  status 
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of  all  capability  deficiencies  associated  with  Block  2B 
fleet  release/Marine  Corps  IOC.  The  review  showed  that 
1,151  reports  were  not  yet  fully  resolved,  151  of  which 
were  assessed  as  “mission  critical”  with  no  acceptable 
workaround  for  Block  2B  fleet  release.  The  remaining 
development  and  flight  test  of  Block  2B  will  determine 
the  final  status  of  these  151  mission  critical  deficiencies, 
whether  they  are  corrected  or  will  add  to  the  incomplete 
development  work  deferred  to  Block  3F  with  the  less 
critical  flaws. 

-  Growth  in  mission  systems  test  points  (regression  for 
new  software  versions,  testing  fixes)  for  CY14  through 
the  end  of  November  was  at  91  percent;  that  is,  for  every 
Block  2B  “baseline”  test  point  accomplished  in  CY14, 

0.91  “growth”  points  have  been  accomplished.  Growth 
in  test  points  for  Block  2B  has  slowed  later  in  CY14  as 
the  program  has  deferred  fixes  of  deficiencies  to  Block  3i 
or  Block  3F,  averaging  61  percent  for  the  period  August 
through  November.  This  average  rate  of  growth,  although 
higher  than  the  planning  rate  for  the  year,  is  less  than  that 
observed  in  CY13  (124  percent)  at  the  time  of  reporting 
for  the  FY13  DOT&E  Annual  Report. 

-  The  program  is  eliminating  test  points  that  are  designed 
to  characterize  performance  (i.e.,  in  a  greater  envelope 
than  a  specific  contract  specification  condition),  reducing 
the  number  of  test  points  needed  to  verify  the  final 
Block  2B  capability  for  fleet  release,  and  deferring  fixes 
for  deficiencies  to  Block  3.  The  program  has  also  added 
points  for  the  capability  required  by  the  Services  to  be 
included  in  Block  2B  capabilities.  Formal  adjustments 
to  the  2014  test  plans  through  the  end  of  October 
resulted  in  a  net  reduction  of  135  Block  2B  baseline  test 
points.  In  November,  the  program  considered  making 
further  adjustments  to  the  plan  in  order  to  complete 
testing  necessary  to  support  Block  2B  fleet  release  by 
the  end  of  January  2015.  After  reviewing  the  remaining 
529  baseline  test  points,  the  program  deemed  139  as 
potentially  no  longer  required  and  another  147  as  optional, 
designating  only  243  of  the  529  remaining  points  as 
essential  for  completing  testing  to  support  Block  2B 

fleet  release.  Formal  adjustments  of  the  test  plans  were 
pending  as  of  the  completion  of  this  report.  These 
reductions  in  the  2014  plan  are  in  addition  to  the  removal 
of  approximately  840  test  points  that  occurred  when  the 
program  consolidated  test  plans  for  software  increments 
prior  to  Block  2B  with  the  plan  for  2014,  all  of  which  were 
planned  to  be  flown  prior  to  the  2014  plan. 

-  The  program  planned  to  complete  Block  2B  mission 
systems  flight  test  in  October,  which  did  not  occur.  The 
completion  date  of  Block  2B  mission  systems  testing 
will  depend,  in  part,  on  realizing  further  reductions  to 
baseline  test  points  and  elimination  of  any  remaining 
restrictions  imposed  on  the  fleet  of  test  aircraft  due  to 
the  engine  failure.  As  of  the  end  of  November,  529  of 
4,183  Block  2B  baseline  test  points  remained.  Assuming 


the  program  would  continue  test  point  productivity  equal 
to  that  realized  in  the  preceding  12  months,  the  program 
will  be  able  to  complete  the  remaining  529  Block  2B  test 
points  by  the  end  of  February  2015.  This  estimate  is  based 
on  the  following  assumptions: 

■  Modifications  to  upgrade  any  additional  mission  systems 
test  aircraft  from  the  Block  2B  to  Block  3i  or  Block  3F 
configuration  (besides  AF-3)  occurs  after  January  2015, 
which  is  the  program’s  current  estimate  for  completing 
Block  2B  development.  Starting  in  February,  two  of  the 
seven  remaining  mission  systems  test  aircraft  upgrade  to 
the  Block  3i  configuration,  while  the  remaining  mission 
systems  test  aircraft  stay  in  the  2B  configuration  to 
complete  testing.  This  schedule  allows  other  mission 
systems  test  aircraft  to  be  modified  to  support  testing  of 
the  Block  3i  and  Block  3F  mission  systems  software, 
the  Generation  III  HMDS,  and  OBIGGS  on  the  F-35C 
variant. 

■  The  operating  restrictions  stemming  from  the  engine 
failure  do  not  restrict  access  to  the  remaining  test  points. 
These  restrictions  are  lifted  on  each  test  aircraft  after  a 
“pre-trenched”  stator  is  installed  in  the  engine.  Through 
the  end  of  November,  the  engines  in  6  of  the  18  test 
aircraft  had  been  modified  with  these  stators  and  the 
program  plans  to  have  the  entire  test  fleet  modified  by 
the  end  of  February  2015. 

■  No  additional  growth  is  experienced  in  the  remainder  of 
Block  2B  flight  testing,  and  deficiencies  not  currently 
addressed  by  fixes  included  in  the  final  test  release  of 
Block  2B  software  (version  2BS5.2)  will  be  deferred  to 
Block  3  or  not  addressed. 

Block  3i 

-  Block  3i  was  not  planned  to  incorporate  any  new 
capability  or  fixes  from  the  Block  2B  development/fleet 
release.  The  first  increment  of  Block  3i  capability, 
designated  3iRl,  is  the  initial  release  to  Lot  6  aircraft  and 
will  include  only  Block  2A  capability  (inherently  less 
capable  than  the  final  Block  2B  fleet  release).  Subsequent 
increments  of  Block  3i  software  will  have  additional 
capability.  However,  the  prospects  for  Block  3i  progress 
are  dependent  on  completion  of  Block  2B  development 
and  flight  test,  which  determines: 

■  When  test  aircraft  are  converted  to  Block  3i;  two  of 
seven  mission  systems  aircraft  -  one  at  the  Edwards 
test  center  and  one  at  the  Patuxent  River,  Maryland,  test 
center  -  have  been  modified  so  far  (flight  testing  can 
only  occur  on  test  aircraft  upgraded  with  TR2  hardware). 

■  How  much  incomplete  development  work  will  be 
inherited  by  Block  3i  due  to  deficiencies  deferred  from 
Block  2B. 

-  Though  it  eventually  began  in  2014,  Block  3i  flight  test 
progress  began  late,  and  has  progressed  much  slower  than 
expected.  As  of  the  end  of  November  2014,  the  program 
had  completed  only  25  percent  of  the  baseline  Block  3i 
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test  points,  accomplishing  177  of  700  test  points,  which 

represented  64  percent  of  the  plan  for  the  year. 

■  The  program  temporarily  modified  two  mission 
systems  aircraft  -  CF-8  in  October  2013  and  AF-3  in 
November  2013  -  with  a  portion  of  the  TR2  hardware 
to  attempt  loading  the  first  build  of  Block  3i  software. 
The  attempt  on  CF-8  failed,  but  the  software  was 
successfully  loaded  on  AF-3,  allowing  the  test  center 
to  complete  ground  software  regression  testing.  AF-3 
was  returned  to  the  Block  2B  configuration  to  support 
testing  until  May  2014,  when  it  underwent  the  full  TR2 
modification  in  preparation  for  Block  3i  flight  testing. 

■  In  May,  the  first  increment  of  flight  test  software  (3iRl) 
was  delivered  to  flight  test  approximately  five  months 
later  than  planned  (December  2013  to  May  2014). 

This  version  of  the  software  is  needed  for  delivery  of 
Lot  6,  TR2  equipped  aircraft.  The  Edwards  test  center 
conducted  flight  testing  of  the  Block  3i  software  on 
AF-3.  The  Patuxent  River  test  center  conducted  one  test 
flight  of  Block  3i  software  on  BF-5,  which  is  currently 
deployed  to  the  climatic  chamber  for  testing.  No  testing 
of  Block  3i  software  has  yet  been  accomplished  on  an 
F-35C  test  aircraft.  As  of  the  end  of  November,  all 
remaining  Block  3i  test  points  were  blocked,  as  the  test 
centers  were  awaiting  the  next  iteration  of  Block  3i 
software  to  proceed  with  flight  testing. 

■  The  test  centers  identified  deficiencies  in  the  3iRl 
software,  five  of  which  needed  to  be  corrected  before  the 
software  could  be  used  in  the  Lot  6  production  aircraft. 
These  deficiencies  were  corrected  and  tested  in  the  lab 
with  an  updated  version  of  software.  This  final  version 
of  3iRl  software  was  not  flight  tested  at  test  centers,  but 
tested  by  the  contractor  at  the  production  facility,  and  is 
used  to  deliver  Lot  6  aircraft. 

■  The  second  iteration  of  Block  3i  software,  3iR4, 
included  capability  to  test  the  new  Generation  III 
HMDS.  The  Edwards  test  center  flew  four  test  missions 
with  3iR4  on  AF-3  in  September,  accomplishing 
regression  test  points  and  some  initial  test  points  from 
the  Generation  III  HMDS  test  plan.  This  was  the  first 
testing  of  the  new  HMDS  on  F-35  test  aircraft.  The 
test  team  discovered  deficiencies,  particularly  in  the 
stability  of  the  new  display  management  computer  for 
the  helmet,  and  suspended  further  testing  until  software 
that  fixes  the  deficiencies  in  the  helmet  system  can  be 
provided  to  the  major  contractor  and  included  in  an 
updated  load  of  mission  systems  software. 

■  The  third  increment  of  Block  3i  software,  version  3iR5, 
will  be  used  to  provide  production  software  for 

Lot  7  aircraft,  the  first  lot  to  be  delivered  with  the 
Generation  III  HMDS.  The  program  plans  for  the 
production  software  to  have  the  equivalent  capabilities 
as  Block  2B  and  plans  to  deliver  3iR5  software  to  flight 
test  in  January  2015.  However,  even  if  this  occurs, 
since  Block  2B  development  and  flight  testing  were 
not  completed  as  planned  in  October,  the  completion 


of  Block  3i  testing  will  be  delayed  if  the  equivalent 
capabilities  from  Block  2B  development  are  to  be 
realized  in  Block  3i.  The  program  plans  to  convert  four 
of  the  five  Block  2B  mission  systems  test  aircraft  at 
the  Edwards  test  center  to  the  Block  3i  configuration  in 
February  2015.  Assuming  this  transition  takes  place, 
Block  3i  flight  testing  could  conclude  by  July  2015, 
two  months  later  than  the  planned  completion  of 
May  2015.  This  assumes  nominal  growth  of  66  percent 
is  experienced  during  the  rest  of  Block  3i  development 
and  flight  testing,  the  program  completes  testing  of 
the  remaining  baseline  test  points  without  reductions, 
and  the  program  uses  four  of  the  six  mission  systems 
test  aircraft  at  the  Edwards  test  center  for  dedicated 
Block  3i  testing.  Of  the  two  remaining  mission  systems 
test  aircraft,  one  other  test  aircraft  could  be  available 
for  further  Block  2B  testing  and  one  could  be  used  to 
start  Block  3F  testing.  Additional  time  will  be  needed 
to  address  corrections  if  additional  deficiencies  are 
identified  in  the  Generation  III  HMDS  and  will  add  risk 
to  the  schedule. 

•  Block  3F 

-  In  order  to  manage  and  complete  Block  3F  development 
and  flight  testing  as  planned  in  late  20 1 7,  the  program 
needs  to  complete  Block  2B  development  and  flight  test  as 
soon  as  possible  and  transition  to  Block  3.  The  program 
currently  acknowledges  four  to  six  months  “pressure” 

on  the  end  of  Block  3F  development  and  test.  The 
program  needs  to  complete  Block  2B  development  soon 
to  focus  resources  (staffing,  labs,  flight  test  aircraft)  on 
the  development  and  testing  of  Block  3F,  designated  as 
“full  warfighting  capability.” 

-  The  test  centers  and  contractor  began  detailed  test 
planning  for  Block  3F  flight  test.  The  draft  test  plan  has 
nearly  6,000  test  points.  Plans  completed  after  the  2012 
re-baselining  of  the  program  showed  the  start  of  Block  3F 
flight  testing  in  May  2014;  however,  current  program  plans 
are  to  start  Block  3F  flight  test  in  March  2015,  10  months 
later  than  the  2012  baseline. 

Mission  Data  Load  Development  and  Testing 

•  The  F-35  relies  on  mission  data  loads  -  which  are  a 
compilation  of  the  mission  data  files  needed  for  operation  of 
the  sensors  and  other  mission  systems  components  -  working 
in  conjunction  with  the  system  software  data  load  to  drive 
sensor  search  parameters  and  to  identify  and  correlate  sensor 
detections  of  threat  radar  signals.  An  initial  set  of  files  was 
produced  by  the  contractor  for  developmental  testing  during 
SDD,  but  the  operational  mission  data  loads  -  one  for  each 
potential  major  area  of  operation  -  will  be  produced  by  a 
U.S.  government  lab,  the  U.S.  Reprogramming  Lab  (USRL). 
These  mission  data  loads  will  be  used  for  operational  testing 
and  fielded  aircraft,  including  the  Marine  Corps  IOC  aircraft. 

•  In  accordance  with  the  approved  mission  data  optimization 
operational  test  plan,  mission  data  loads  undergo  a 
three-phased  lab  development  and  test  regimen,  followed  by 
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flight  test.  The  current  plans  are  to  certify  the  first  two  mission 
data  loads,  which  are  needed  to  support  Marine  Coips  IOC, 
in  November  2015  after  flight  testing  occurs  on  operational 
test  aircraft  between  March  and  October  2015.  These  plans 
provide  the  mission  data  load  later  than  needed  for  the 
Marine  Corps’  objective  IOC  date  of  July  2015.  However, 
truncating  the  mission  data  load  development  and  conducting 
open-air  flight  testing  early  on  a  limited  open-air  range  for 
the  purpose  of  releasing  a  mission  data  load  in  mid-2015 
would  create  significant  operational  risk  to  fielded  units, 
since  the  load  will  not  have  completed  the  planned  lab  testing 
and  because  the  open-air  range  test  infrastructure  is  capable 
of  verifying  only  a  small  portion  of  the  mission  data.  The 
program  should  complete  lab  testing  of  the  mission  data  loads, 
as  is  planned  in  the  mission  data  optimization  operational 
test  plan,  prior  to  accomplishing  the  necessary  flight  testing 
to  ensure  the  loads  released  to  the  fleet  are  optimized  for 
performance.  If  mission  data  loads  are  released  to  operational 
units  prior  to  the  completion  of  the  lab  and  flight  testing 
required  in  the  operational  test  plan,  the  risk  to  operational 
units  must  be  clearly  documented. 

•  Several  items  are  currently  creating  risk  to  the  program’s 
ability  to  deliver  certified  mission  data  loads.  Mission  data 
lab  equipment  was  held  by  the  major  contractor  at  their 

Fort  Worth  facility  for  three  years  past  the  planned  delivery  to 
the  USRL  to  support  mission  systems  software  development 
for  production  aircraft,  reducing  productivity  at  the  USRL. 

The  USRL  did  not  receive  sufficient  documentation  of 
the  equipment  and  software  tools  that  were  delivered  by 
the  program;  this  has  hampered  their  training  and  slowed 
development.  Contract  issues  had  prevented  USRL  from 
direct  communications  with  the  subcontractor  that  designed 
both  the  electronic  warfare  system  on  the  aircraft  and  the 
mission  data  programming  tools.  These  communications  were 
needed  to  understand  undocumented  lab  and  mission  data 
file  generation  tool  functions.  The  Program  Office  has  taken 
steps  to  improve  these  communications.  Other  challenges  that 
may  affect  on-time  delivery  of  mission  data  include  instability 
in  the  contractor-delivered  mission  data  file  generation  tool, 
which  creates  the  final  mission  data  load,  and  slower  than 
expected  development  of  software  analysis  tools  that  optimize 
sensor  performance. 

•  Mission  data  load  development  and  testing  is  a  critical  path 
to  combat  capability  for  Block  2B  and  Block  3F.  Accuracy 
of  threat  identification  and  location  depend  on  how  well  the 
mission  data  loads  are  optimized  to  perform  in  ambiguous 
operational  environments.  This  is  difficult  work  given  a  stable 
software  capability  in  the  platform,  adequate  lab  equipment, 
and  stable/well-understood  mission  data  file  generation 

tools  -  none  of  which  are  yet  available  in  the  program. 

•  The  current  lab  is  essentially  a  copy  of  the  mission  systems 
integration  lab  used  by  the  major  contractor  to  integrate  and 
test  software.  It  is  not  adequate  for  development  of  mission 
data  loads  for  use  in  operationally  realistic  conditions.  As 
identified  by  DOT&E  in  early  2012,  the  program  must  plan 


and  execute  a  significant  upgrade  to  the  lab  in  order  for  it 
to  generate  an  operationally  realistic  signal  environment  for 
mission  data  load  optimization.  Though  funding  has  been 
made  available,  plans  for  this  upgrade,  and  integration  with  the 
Block  2B,  Block  3i,  and  Block  3F  mission  data  loads  have  not 
been  finalized. 

Weapons  Integration 

•  Progress  in  weapons  integration,  in  particular  the  completion 
of  planned  weapon  delivery  accuracy  (WDA)  events,  has  been 
very  limited  in  2014  compared  to  that  planned  by  the  program. 
Multiple  deficiencies  in  mission  systems,  aircraft  grounding, 
and  subsequent  flight  restrictions  caused  by  the  June  engine 
failure  all  contributed  to  the  limited  progress. 

•  Each  WDA  event  requires  scenario  dry-runs  in  preparation 
for  the  final  end-to-end  event  to  ensure  the  intended  mission 
system  functionality,  as  well  as  engineering  and  data  analysis 
requirements  (to  support  the  test  centers  and  weapon  vendors) 
are  available  to  complete  the  missile  shot  or  bomb  drop.  Per 
the  approved  TEMP,  these  preparatory  events,  as  well  as  the 
end-to-end  events,  are  to  be  accomplished  with  full  mission 
systems  functionality,  including  operationally  realistic  fire 
control  and  sensor  performance. 

•  Mission  systems  developmental  testing  of  system  components 
required  neither  operation  nor  full  functionality  of  subsystems 
that  were  not  a  part  of  the  component  under  test.  The 
individual  mission  system  component  tests  were  designed 

by  the  developmental  teams  to  verify  compliance  with 
contract  specification  requirements  rather  than  to  test  the  full 
mission  systems  performance  of  the  aircraft  and  complete  the 
find-fix-ID-track-target-engage-assess  kill  chain  for  air-to-air 
and  air-to-ground  mission  success.  WDA  events,  however, 
were  specifically  designed  to  gather  both  the  necessary 
weapons  integration  and  fire-control  characterization  and 
performance  using  all  the  mission  systems  required  to  engage 
and  kill  targets. 

•  Planning  and  scheduling  of  the  WDA  events  assumed  that  all 
associated  mission  systems  functionality  would  be  mature 
by  the  WDA  preparatory  event  dates.  However,  due  to  the 
limitations  in  progress  in  Block  2B  mission  systems,  this  has 
not  occurred. 

-  Deficiencies  in  the  Block  2B  mission  systems  software 
affecting  the  WDA  events  were  identified  in  fusion, 
radar,  passive  sensors,  identification  friend-or-foe, 
electro-optical  targeting  system,  and  the  aircraft  navigation 
model.  Deficiencies  in  the  datalink  systems  also  delayed 
completion  of  some  events.  Overall,  these  deficiencies 
have  both  delayed  the  WDA  event  schedule  and 
compromised  the  requirement  to  execute  the  missions  with 
fully  functional  and  integrated  mission  systems. 

-  The  program  had  planned  to  complete  all  Block  2B  WDA 
events  by  October  2014.  This  did  not  occur.  Through 
the  end  of  November,  10  of  15  live  fire  events  had  been 
completed,  while  the  program  planned  to  have  all  15 
completed  by  the  end  of  October.  In  November,  the 
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WEAPONS  DELIVERY  ACCURACY  (WDA)  PROGRESS 


Weapon 

WDA 

Number 

Preparatory  Events 

End-to-End  Event 

Planned 

Completed/ 

Scheduled ’ 

Weeks 

Delayed 

Planned 

Completed/ 

Scheduled 

Weeks 

Delayed 

AIM-120 

102 

Sep  13 

Sep  13 

2 

Oct  13 

Oct  13 

2 

112 

Sep  13 

Sep  13 

3 

Oct  13 

Nov  13 

3 

GBU-12 

113 

Sep  13 

Oct  13 

3 

Oct  13 

Oct  13 

0 

GBU-32 

115 

Sep  13 

Nov  13 

6 

Nov  13 

Dec  13 

3 

AIM-120 

108 

Oct  13 

Dec  13 

7 

Dec  13 

Feb  14 

12 

110 

Oct  13 

Aug  13 

43 

Dec  13 

Nov  14 

50 

111 

Dec  13 

Deferred  to 
Block  3F 

-- 

Jan  14 

Deferred  to 
Block  3F 

-- 

106 

Dec  13 

Sep  14 

40 

Jan  14 

Nov  14 

43 

GBU-31 

114 

Dec  13 

May  14 

45 

Feb  14 

Nov  14 

41 

Jun  14 

Oct  14 

AIM-120 

104 

Feb  14 

Aug  14 

30 

Mar  14 

Jan  15 

44 

Sep  14 

107 

Mar  14 

Jun  14 

12 

May  14 

Dec  14 

30 

101 

May  14 

May  14 

17 

Jun  14 

Dec  14 

26 

Sep  14 

103 

Jun  14 

Mar  14 

-8 

Aug  14 

May  14 

-10 

Apr  14 

109 

Jul  14 

Jan  14 

-29 

Sep  14 

Mar  14 

-27 

105 

Sep  14 

Deferred  to 
Block  3  F 

-- 

Oct  14 

Deferred  to 
Block  3F 

-- 

1 .  Some  WDA  events  require  more  than  one  preparatory  event. 


Cracking  in  the  LHS  F2  fuel  floor  flange,  found  at 
11,000  EFH 


program  deferred  two  of 
the  planned  Block  2B  WDA 
events  to  Block  3,  due  to 
deficiencies  and  limitations 
of  capability  in  Block  2B 
mission  systems.  The 
adjacent  table  shows  the 
planned  date,  completion 
or  scheduled  date,  and 
weeks  delayed  as  of  the  end 
of  November  for  each  of 
the  WDA  preparatory  and 
end-to-end  events.  Events 
completed  are  shown  with 
dates  in  bold  font;  events 
scheduled  are  shown 
with  dates  in  italicized 
font.  The  program  should 
complete  the  remaining 
three  Block  2B  WDA 
flight  test  events,  using  the 
currently  planned  scenarios, 
and  ensuring  full  mission 
systems  functionality  is 
enabled  in  an  operationally 
realistic  manner. 


Static  Structural  and  Durability 
Testing 

•  Structural  durability  testing 
of  all  variants  using  full  scale 
test  articles  is  ongoing,  each 
having  completed  at  least  one  full  lifetime  (8,000  equivalent 
flight  hours,  or  EFF1).  All  variants  are  scheduled  to  complete 
three  full  lifetimes  of  testing  before  the  end  of  SDD;  however, 
complete  teardown,  analyses,  and  Damage  Assessment  and 
Damage  Tolerance  reporting  is  not  scheduled  to  be  completed 
until  August  2019.  The  testing  on  all  variants  has  led  to 
discoveries  requiring  repairs  and  modification  to  production 
designs  and  retrofits  to  fielded  aircraft. 

•  F-3  5 A  durability  test  article  (AJ-1)  completed  11,000  EFF1  on 
September  13,  which  is  3,000  hours  into  the  second  lifetime. 
Testing  restarted  on  October  29,  after  completing  non-invasive 
inspections,  which  are  required  at  1,000  EFF1  intervals. 

-  Cracking  of  the  right  hand  side  (RF1S)  Fuselage  Station 
(FS)  402,  discovered  after  the  first  lifetime  of  testing 
(8,000  EFF1)  at  the  end  of  CY12,  required  repairs  to  the  test 
article,  production  redesign  for  production  Lot  8  and  later 
aircraft,  and  retrofitting  a  modification  for  production  Lot  4 
through  7  aircraft. 

-  Discoveries  from  the  second  lifetime  of  testing,  which 
started  on  December  13,  2013,  include: 

■  Cracking  of  the  left  hand  side  (LF1S)  integrated  power 
package  shear  web  lug  at  FS503,  found  at  10,082  EFF1 

■  Cracking  of  the  LF1S  FS503  frame  support,  found  at 
10,162  EFH 


-  Disposition  of  these  discoveries  and  repair  plans  were 
under  consideration  as  of  the  time  of  this  report. 

F-35B  durability  test  article  (BH-1)  has  been  halted  since 
September  2013,  when  the  FS496  bulkhead  severed 
at  9,056  EFH,  transferred  loads  to  an  adjacent  FS518 
bulkhead,  and  caused  cracking.  Root  cause  analysis  and 
corrective  action  -  for  repairing  the  bulkheads  on  the  test 
article,  modification  for  the  fielded  aircraft,  and  redesign 
for  production  Lot  8  (and  subsequent  lots)  -  have  been 
ongoing  throughout  CY14.  The  program  planned  to  restart 
testing  in  late  September  2014,  but  repairs  took  longer  than 
expected.  Testing  had  not  restarted  as  of  the  end  of  November. 
According  to  the  Program  Office,  the  effect  on  fielded  aircraft 
will  be  limited  life  for  FS496  (approximately  10  years  of 
service  life)  until  replaced  or  repaired. 

-  Modifications  to  the  test  article  include  the  addition  of 
seven  splice  plates  to  repair  cracks  in  the  FS496  and  FS518 
bulkheads. 

-  For  retrofitting/modifying  FS496  in  production  aircraft 
in  Lots  1  through  8,  the  program  is  considering  a  number 
of  fatigue  mitigation  steps,  including  relocating  system 
attachment  points,  hardening  the  fastener  holes  through  a 
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cold  working  process,  and  the  use  of  laser  shock  peening 
(LSP)  to  enhance  fatigue  life  in  sections  of  the  bulkhead 
where  tensile  stresses  are  known  to  be  concentrated. 

The  objective  of  treating  areas  with  LSP  is  to  create 
compressive  pre-stress  states  near  surfaces  where  tensile 
stresses  are  expected  to  be  high  and  hence  reduce  crack 
initiation.  However,  LSP  has  not  been  used  on  the  type 
of  aluminum  alloy  (AL-7085)  used  in  manufacturing  the 
FS496  bulkheads  in  the  F-35B,  and  the  ability  to  affect  the 
structural  life  is  not  well  understood.  The  program  should 
require  the  contractor  to  conduct  rigorous  finite-element 
analyses  to  assess  the  benefit  of  LSP  application.  The 
main  objectives  are  to  assess  the  LSP  effect  in  reducing 
tensile  stress  concentrations  in  critical  areas  and  to  assure 
limited  increase  of  tensile  stresses  in  the  other  areas.  To 
date,  the  effect  on  AL-7085  fatigue  properties  due  to 
LSP  application  are  yet  to  be  characterized,  therefore  a 
finite-element  analysis  using  the  existing  AL-7085  fatigue 
property  data  is  likely  to  over-estimate  the  effect  of  LSP  in 
improving  fatigue  resistance,  which  should  also  be  taken 
into  account. 

-  For  aircraft  in  Lot  9  and  beyond,  the  program  is 
redesigning  the  five  carry-through  bulkheads  in  the  F-35B 
(FS450,  FS472,  FS496,  FS518,  and  FS556).  The  redesign 
will  include  LSP  on  two  bulkheads,  cold  working  of 
fastener  holes  on  four,  and  increasing  thickness  in  portions 
of  all  five  bulkheads.  The  overall  effect  on  aircraft  weight 
increase  is  not  yet  known. 

-  Because  of  the  extensive  repair  required  to  the  FS496 
bulkhead,  the  certification  path  to  full  life  will  likely 
require  additional  follow-on  testing. 

•  F-35C  durability  test  article  (CJ-1)  began  second  lifetime 
testing  on  April  2,  and  completed  2,312  EFH  into  the 
second  lifetime  in  August  (10,312  EFH  total),  followed  by 
inspections.  Testing  resumed  October  28,  2014. 

-  Discoveries  after  the  first  lifetime  of  testing  caused 
redesigns  in  the  FS518  fairing  support  frame  and  FS402 
upper  inboard  frame.  Repairs  and  redesigns  were 
completed  at  8,869  EFH  and  8,722  EFH,  respectively. 

-  Discoveries  from  the  second  lifetime  of  testing  include 
cracking  of  outboard  wing  spar  #5  and  cracking  on  both  the 
left  and  right  hand  sides  of  the  FS575  center  arch  frame. 
Repairs  to  both  were  completed  at  10,000  EFH  prior  to 
restart  of  testing. 

Modeling  and  Simulation 

Verification  Simulation  (VSim) 

•  The  Verification  Simulation  (VSim)  is  a  man-in-the-loop, 
mission  software-in-the-loop  simulation  developed  to  meet 
the  OTAs’  requirements  for  Block  3F  IOT&E,  as  well  as  to 
provide  a  venue  for  contract  compliance  verification  for  the 
Program  Office. 

•  At  the  beginning  of  C Y 1 4,  the  program  planned  to 
accredit  the  VSim  for  use  in  Block  2B  contract  compliance 
verification  by  the  end  of  the  year.  However,  lack  of 


progress  on  the  Verification  and  Validation  (V&V)  process, 
and  to  a  lesser  extent  the  VSim  development  process,  caused 
the  program  to  charter  an  independent  review  of  VSim. 

This  review  eventually  led  to  cancellation  of  the  contract 
verification  portion  of  Block  2B  VSim  planned  usage. 

For  similar  reasons,  after  the  Block  2B  OUE  re-scoping 
effort  began,  the  JSF  Operational  Test  Team  determined 
that  VSim  would  likely  not  support  planned  Block  2B 
operational  testing  in  2015  and  reduced  the  requirements 
for  the  simulation’s  intended  uses  to  support  only  tactics 
development  and  other  activities  that  directly  contribute  to 
the  fielding  of  Block  2B  capabilities. 

•  About  one-third  of  the  validation  evidence  for  Block  2B 
VSim  was  reviewed  by  the  developmental  and  operational 
test  stakeholders  before  the  contractual  use  of  VSim  for 
Block  2B  was  cancelled.  This  review  confirmed  that 
additional  time  was  needed  before  VSim  V&V  could 
potentially  meet  expectations.  Collaborative  replanning 
of  Block  2B  activities  is  not  complete,  but  V&V  reviews 
to  support  operational  testing  needs  are  now  planned 

for  early  2015,  with  accreditation  of  VSim  for  tactics 
development  and  other  uses  expected  in  October  2015. 

•  Exercising  the  V&V  process  for  Block  2B  VSim  is  critical 
to  reducing  risk  for  its  use  in  Block  3F  IOT&E.  Rigorous 
validation  will  identify  gaps  in  VSim  performance,  including 
threat  modeling,  in  time  to  create  the  appropriate  fixes  for 
Block  3F.  Creation  of  test  and  V&V  procedures  as  well  as 
V&V  reports  and  accreditation  documentation  will  provide  a 
significantly  better  understanding  of  VSim  status  by  the  end 
of  2015. 

•  Rigorous  validation  depends  on  good  source  data,  and  the 
contractor  and  Program  Office  improved  efforts  to  ensure 
VSim  needs  are  met  in  the  Block  3F  flight  test  plan.  Those 
plans  are  not  finalized,  but  will  certainly  result  in  deficits  as 
the  enterprise-wide  need  for  flight  tests  exceeds  available 
resources.  Success  in  validating  Block  3F  VSim  will  depend 
on  bridging  this  gap  with  acceptable  data  sources. 

•  The  contractor  has  increased  resources  on  VSim  V&V  teams, 
and  the  quality  of  the  V&V  products  is  increasing.  However, 
the  rate  of  completing  validation  points  (a  comparison  of 
VSim  model  performance  to  aircraft  hardware  performance 
under  similar  test  conditions  using  data  from  flight  test, 
avionics  test  bed,  or  labs),  has  been  much  slower  than 
planned.  This  makes  completing  the  validation  reports, 
which  analyze  the  points  with  respect  to  intended  use,  at  risk 
to  support  even  the  reduced  accreditation  requirements  for 
Block  2B.  Additional  resources  may  be  required  to  complete 
the  significant  task  of  validating  the  complex  federation  of 
models  in  VSim  in  time  for  Block  3F  IOT&E. 

•  Although  the  VSim  validation  process  has  improved, 
DOT&E  has  continued  to  highlight  shortfalls  in  the  test 
resources  needed  to  gather  key  elements  of  data  required 
for  validation  of  the  VSim  for  IOT&E,  in  particular  for 
electronic  warfare  performance  in  the  presence  of  advanced 
threats.  These  shortfalls  are  a  function  of  limitations  in  the 
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test  assets  currently  available  to  represent  threat  systems. 
DOT&E  has  made  formal  recommendations  to  address 
the  shortfalls  and  is  pursuing  solutions  to  make  the  assets 
available  in  time  to  prepare  for  IOT&E  in  a  realistic  threat 
environment. 

•  Limiting  VSim  Block  2B  validation,  and  use,  to  tactics 
development  and  evaluation  will  help  the  program  progress 
towards  V&V  of  Block  3F.  Block  3F  use  of  VSim  for 
IOT&E  is  not  optional;  it  is  required  for  an  adequate  IOT&E. 

Training  System 

•  Pilot  training  continues  at  Eglin  AFB,  Florida,  and  expanded 
in  September  2014  when  additional  F-35B  training  began 

at  MCAS  Beaufort,  South  Carolina.  Additional  F-35A  pilot 
training  is  planned  to  start  in  May  2015  at  Luke  AFB,  Arizona. 
Sixty-six  student  pilot  training  slots  were  available  in  FY14, 
but  nine  were  not  used  due  to  reduced  Service  requirements. 

•  The  training  center  began  transitioning  from  the  Block  IB 
to  the  Block  2A  training  syllabus  for  all  three  variants 

in  December  2013,  and  completed  the  transition  in 
February  2014.  The  ability  to  train  in  and  for  adverse  weather 
conditions  was  added  to  the  Block  2A  syllabus  during  CY14. 
The  Block  2B  syllabus  is  planned  for  delivery  in  mid-2015, 
and  is  planned  to  include  limited  combat  capability. 

•  Lot  5  deliveries  to  pilot  training  bases  continued  throughout 
2014,  including  the  first  nine  F-35A  to  Luke  AFB,  and 

an  additional  eight  F-35A,  one  F-35B,  and  six  F-35C 
aircraft  to  Eglin  AFB.  Lot  6  deliveries,  which  began  in  late 
October,  included  the  first  F-35B  aircraft  delivered  directly 
to  MCAS  Beaufort  where  it  joined  other  F-35B  aircraft 
transferred  from  Eglin  AFB. 

•  All  training  to  date  has  been  in  Block  2A-configured  aircraft, 
which  have  envelope  and  other  restrictions  that  preclude 
high  performance  training  events.  Because  of  this,  all  pilots 
attending  Block  2A  training  complete  only  a  portion  of  the 
planned  syllabus  before  moving  to  their  units. 

•  The  Training  Management  System  (TMS)  is  a  database  that 
includes  course  material,  syllabus  flow,  student  records, 
and  schedules  for  aircrew  and  maintainers.  The  academic 
center  is  using  the  TMS  for  instruction  and  tracking  student 
progress.  TMS  functionality  is  relatively  unchanged  from 
that  which  existed  during  the  2012  training  system  OUE. 

For  example,  the  TMS  cannot  yet  be  effectively  used  for 
scheduling,  pilot  qualification  tracking,  and  the  other  typical 
unit  functions.  This  year,  the  Program  Office  added  funding 
to  correct  these  deficiencies  and  improve  the  functionality 
for  tracking  operational  unit  “continuation  training,” 
which  includes  monthly  training  requirements  and  pilot 
qualifications.  Planned  delivery  is  in  the  2017  timeframe, 
and  will  also  require  Automatic  Logistics  Information  System 
(ALIS)  system-level  architecture  modifications  to  achieve 
full  capability.  Until  then,  flying  units  at  both  training  and 
operational  bases  will  most  likely  continue  to  use  legacy 
scheduling  and  training  databases,  which  causes  double  entry 
into  databases  and  impedes  program-level  data  analysis  such 
as  annual  flying  hour  progress. 


•  The  training  center  continued  to  conduct  maintenance 
training  for  experienced  maintenance  personnel  for  all 
F-35  variants  during  2014.  As  of  the  end  of  October,  more 
than  1,800  U.S.  personnel  and  foreign  partner  students 
had  completed  training  in  one  or  more  of  the  maintenance 
courses,  including  ALIS,  to  support  fielded  maintenance 
operations.  For  the  12-month  period  ending  in  October  2014, 
the  contractor  provided  1,018  training  slots  for  maintenance 
courses,  of  which  701  were  filled  by  U.S.  or  foreign  partner 
students,  equating  to  69  percent  training  seat  utilization  rate. 

In  addition,  active  duty  personnel  at  the  field  units  conducted 
training  that  is  not  included  in  these  metrics.  The  Integrated 
Training  Center  at  Eglin  AFB  currently  offers  13  maintenance 
classes  ranging  from  3  to  13  weeks  in  length. 

Live  Fire  Test  and  Evaluation 

F-35B  Full-Scale  Structural  System  Vulnerability  Assessment 

•  The  F-35  LFT&E  Program  completed  the  F-35B  full-scale 
structural  test  series.  The  Navy’s  Weapons  Survivability 
Laboratory  (WSL)  in  China  Lake,  California,  completed 
15  tests  events  using  the  BG:0001  test  article.  Preliminary 
review  of  the  results  indicates  that: 

-  Anti-aircraft  artillery  (AAA)  threat-induced  damage 
stressed  the  critical  wing  structure  members,  but  multiple 
structural  load  paths  successfully  limited  the  damage  to 
expected  areas  around  the  impact  points  while  preserving 
the  static  flight  load  carrying  capabilities.  Consistent 
with  predictions,  the  tests  demonstrated  other  cascading 
damage  effects,  including  threat-induced  fire  and  damage 
to  adjacent  fuselage  fuel  tanks. 

-  AAA  and  missile  fragment-induced  damage  stressed 
the  structural  limitations  of  the  forward  fuselage  fuel 
tanks  (F-l  and  F-2).  Cascading  effects  from  the  F-l  tank 
damage  included  a  large  fuel  release  into  the  cockpit  and 
damage  to  the  pilot  seat  mounting  structure.  To  mitigate 
the  vulnerability  to  the  pilot,  the  Program  Office  has 
recently  altered  the  F-35B  fuel  burn  strategy  so  that  the 
F-l  tank  behind  the  pilot  empties  sooner.  Threat-induced 
damage  in  these  fuel  tank  tests  also  caused  large  fuel 
discharge  into  the  engine  inlet,  which  would  have  likely 
caused  engine  failures  due  to  fuel  ingestion.  The  engine 
was  not  installed  for  these  tests. 

-  The  extent  of  AAA-induced  structural  damage  to  the  wing 
leading  edge  flap  and  the  horizontal  tail  is  not  flight  critical 
from  a  structural  tolerance  perspective.  The  leading  edge 
tests  demonstrated  the  potential  for  sustained  fire,  which 
could  have  flight-critical  cascading  effects  on  the  wing 
structure. 

-  The  ballistic  damage  tolerance  testing  of  propulsion 
system  related  structural  components  (variable  area  vane 
box  nozzle,  and  hinges  on  the  roll  duct  nozzle,  lift  fan,  and 
auxiliary  air  inlet  doors)  revealed  these  components  were 
nearly  insensitive  to  expected  threats.  However,  sustained 
fires  were  created  in  the  shot  into  the  variable  area  vane 
box  nozzle  due  to  leakage  in  the  actuating  hydraulics,  and 
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the  shot  into  the  roll  duct  nozzle  door  due  to  damage  to 
the  adjacent  fuel  tank.  These  fires  would  ultimately  have 
led  to  cascading  structural  damage. 

-  Data  support  the  evaluation  of  residual  loading 
capabilities  of  the  aft  boom  structure,  including 
vertical  tail  and  horizontal  tail  attachments,  following  a 
man-portable  air  defense  system  impact  and  detonation. 
While  having  fuel  in  the  aft-most  F-5  fuel  tank  increased 
structural  damage  due  to  resultant  hydrodynamic  ram 
effects  and  fire,  flight  control  surfaces  remained  attached. 
Further  structural  analysis  of  the  damage  effects  is  being 
completed  to  verify  the  structural  integrity  of  the  aft  boom 
structure. 

FI  35  Engine 

•  F135  live  fire  engine  testing  in  FY13,  engine  vulnerability 
analysis  in  FY13,  and  uncontained  engine  debris  damage 
analysis  in  FY03  demonstrated  two  primary  threat-induced 
engine  damage  mechanisms: 

-  Penetration  of  the  engine  case  and  core  that  could  cause 
blade  removal,  resulting  in  damage  to  turbomachinery 
leading  to  propulsion  loss  or  fire 

-  Damage  to  external  engine  components  (e.g.,  fuel  lines, 
pumps,  gearbox,  etc.)  leading  to  critical  component 
failure  and  fire 

•  Engine  fuel  ingestion  testing  in  FY07  further  demonstrated 
the  potential  of  an  engine  stall  providing  a  fire  ignition 
source  in  the  presence  of  additional  fuel  system  damage. 

•  The  uncontained  FI 35  fan  blade  release  and  subsequent  fuel 
fire  in  an  F-35A  at  Eglin  AFB  in  June  provides  an  additional 
data  point  that  needs  to  be  reviewed  and  analyzed  to  support 
the  F-35  vulnerability  assessment. 

Polyalphaolefin  (PAO)  Shut-Off  Valve 

•  The  Program  Office  tasked  Lockheed  Martin  to  develop  a 
technical  solution  for  a  PAO  shut-off  valve  to  meet  criteria 
developed  from  live  fire  test  results.  This  aggregate, 
2-pound  vulnerability  reduction  feature,  if  installed,  would 
reduce  the  probability  of  pilot  incapacitation,  decrease 
overall  F-35  vulnerability,  and  prevent  the  program  from 
failing  one  of  its  vulnerability  requirements. 

•  The  program  has  not  provided  any  updates  on  the  feasibility 
and  effectiveness  of  the  design,  nor  an  official  decision  to 
reinstate  this  vulnerability  reduction  feature. 

Fuel  Tank  Ullage  Inerting  System 

•  The  program  verified  the  ullage  inerting  design  changes 
and  demonstrated  improved,  inerting  performance  in  the 
F-35B  fuel  system  simulator  (FSS)  tests.  A  preliminary  data 
review  demonstrated  that  the  system  pressurized  the  fuel 
tank  with  nitrogen  enriched  air  (NEA)  while  maintaining 
pressure  differentials  within  design  specifications  during 

all  mission  profiles  in  the  simulator,  including  rapid  dives, 
but  revealed  the  potential  for  pressure  spikes  from  the 
aerial  refueling  manifold,  as  noted  in  the  flight  sciences 
section  of  this  report.  The  Program  Office  will  complete 
and  document  detailed  data  review  and  analyses  to  evaluate 


NEA  distribution  and  inerting  uniformity  between  different 
fuel  tanks  and  within  partitioned  fuel  tanks. 

•  The  program  developed  a  computational  model  to  predict 
inerting  performance  in  the  aircraft  based  on  the  F-35B 
simulator  test  results.  Patuxent  River  Naval  Air  Station 
completed  the  ground  inerting  test  on  an  actual  F-35B 
aircraft  to  verify  the  model,  but  a  detailed  comparison  to 
F-35B  FSS  has  not  yet  been  completed.  The  program  will 
use  this  model,  in  conjunction  with  the  completed  F-35  A 
ground  tests  and  F-35C  ground  tests  planned  to  start  in 
February  2015,  to  assess  the  ullage  inerting  effectiveness 
for  all  three  variants.  The  confidence  in  the  final  design  and 
effectiveness  will  have  to  be  reassessed  after  the  deficiencies 
uncovered  in  the  aircraft  ground  and  flight  tests  have  been 
fully  resolved. 

•  When  effective,  ullage  inerting  only  protects  the  fuel  tanks 
from  lightning-induced  damage.  The  program  has  made 
progress  in  completing  lightning  tolerance  qualification 
testing  for  line-replaceable  units  needed  to  protect  the 
remaining  aircraft  systems  from  lightning-induced  currents. 
Lightning  tolerance  tests  using  electrical  current  injection 
tests  are  ongoing,  and  the  program  is  expected  to  complete 
the  tests  by  2QFY 15. 

Electrical  System 

•  DOT&E  expressed  a  concern  in  FY13  for  the  potential  loss 
of  aircraft  due  to  ballistically-induced  shorting  of  power  and 
control  circuits  in  the  F-35  flight  control  electrical  systems. 
The  F-35  is  the  first  tactical  fighter  aircraft  to  incorporate  an 
all-electric  flight  control  system,  using  a  270  Volt  power  bus 
to  power  flight  control  actuator  systems  and  a  28  Volt  bus 

to  control  those  actuators.  The  F-35  aircraft  carries  these 
voltages  in  wire  bundles  where  they  are  in  close  proximity. 
Live  fire  tests  of  similar  wire  bundle  configurations 
demonstrated  the  potential  for  arcing  and  direct  shorts  due  to 
ballistic  damage. 

•  Lockheed  Martin  completed  an  electrical  power  systems 
report,  which  included  a  summary  of  development  tests 
to  demonstrate  that  transient-voltage  suppression  diodes 
installed  throughout  the  28  Volt  systems  shunt  high  voltage 
(including  270  Volt)  to  ground,  preventing  the  high  voltage 
from  propagating  to  other  flight-critical  components.  Some 
components  might  be  damaged  as  a  result  of  a  short,  but  their 
redundant  counterparts  would  be  protected.  Testing  used 
direct  injection  of  the  high  voltage,  rather  than  shorting  from 
ballistic  damage,  but  the  electrical  effects  would  be  the  same. 

Vulnerability  to  Unconventional  Threats 

•  The  full-up,  system-level  chemical-biological 
decontamination  test  on  the  BF-4  test  article  planned  for 
4QFY16  at  Edwards  AFB  is  supported  by  two  risk-reduction 
events: 

-  The  Limited  Demonstration  event  conducted  in  4QFY14 
showed  that  the  proposed  decontamination  shelter  and 
liner  design  can  sustain  conditions  of  160°F  and  80  percent 
relative  humidity.  The  high  temperature  alone  is  sufficient 
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to  decontaminate  chemical  agents.  The  combination 
of  high  heat  and  humidity  has  been  shown  effective  in 
decontaminating  biological  agents.  Both  chemical  and 
biological  decontamination  techniques  take  10  to  12  days  to 
complete. 

-  A  System  Integration  Demonstration  of  the 

decontamination  equipment  and  shelter  was  conducted 
on  an  F- 16  test  article  during  1QFY15  at  Edwards  AFB 
to  simulate  both  hot  air  chemical  and  hot/humid  air 
biological  decontamination  operations.  This  testing  will  not 
demonstrate  the  decontamination  system  effectiveness  in  a 
range  of  operationally  realistic  environments. 

•  The  F-35  variant  of  the  Joint  Service  Aircrew  Mask 
(JSAM-JSF)  successfully  passed  its  Preliminary  Design 
Review  in  3QFY14.  The  Joint  Program  Executive  Office 
for  Chemical  and  Biological  Defense  and  the  F-35  Program 
Office  will  have  to  integrate  the  JSAM-JSF  with  the 
Flelmet-Mounted  Display,  which  is  undergoing  a  challenging 
design  process  and  consequently  further  aggravating  this 
integration  effort. 

•  Planned  EMP  testing  will  evaluate  the  aircraft  to  the  threat 
level  defined  in  MIL-STD-2169B.  Both  horizontal  and 
vertical  polarization  testing,  as  well  as  active,  passive,  and 
direct  drive  testing  are  planned  to  assess  effects  and/or 
damage  of  the  EMP  induced  currents  and  coupling  to  vehicle 
and  mission  systems  electronics.  EMP  testing  on  the  F-35B 
article  was  completed  in  1QFY15;  data  analysis  is  ongoing. 
Follow-on  tests  on  other  variants,  including  a  test  series 

to  evaluate  any  Block  3F  hardware/software  changes,  are 
planned  for  FY16. 

Gun  Ammunition  Lethality  and  Vulnerability 

•  The  program  completed  the  ballistic  impact  response 
characterization  of  the  PGU-47/U  Armor  Piercing  Explosive 
(APEX)  round  for  the  partner  F-35  A  variant  using  the  AAA 
and  fragment  threats.  Preliminary  data  analysis  demonstrated 
no  significant  reactions  or  evidence  of  high  pressures  that 
could  potentially  induce  sympathetic  reactions  from  adjacent 
rounds  loaded  on  the  aircraft. 

•  The  program  completed  the  terminal  ballistic  testing  of 
the  PGU-48  FAP  round  and  the  PGU-32  round  against 

a  range  of  target-representative  material  plates  and  plate 
arrays.  Preliminary  FAP  test  observations  indicate  lower 
than  expected  levels  of  fragmentation  when  passing  through 
multiple  layer  targets.  PGU-32  test  observations  indicate 
that  the  round  detonates  much  closer  to  the  impact  point  of 
the  first  target  plate  than  originally  called  out  in  ammunition 
specification.  The  program  will  determine  the  impact  of  these 
data  on  the  ammunition  lethality  assessment. 

•  Ground-based  lethality  test  planning  is  ongoing.  All  three 
rounds  will  be  tested  against  a  similar  range  of  targets, 
including  armored  and  technical  vehicles,  aircraft,  and 
personnel  in  the  open.  FY15  funds  are  in  place  for  all  tests 
except  those  against  boat  targets. 

•  Air-to-ground  lethality  tests  will  likely  begin  no  earlier  than 
1QFY16.  Given  the  development  test  schedule  of  the  APEX 
round,  the  existing  flight  test  plan  does  not  include  this  round. 
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Operational  Suitability 

•  Overall  suitability  continues  to  be  less  than  desired  by 
the  Services,  and  relies  heavily  on  contractor  support  and 
unacceptable  workarounds,  but  has  shown  some  improvement 
in  CY14. 

-  Aircraft  availability  was  flat  over  most  of  the  past  year, 
maintaining  an  average  for  the  fleet  of  37  percent  for  the 
12-month  rolling  period  ending  in  September  -  consistent 
with  the  availability  reported  in  the  FY13  DOT&E 
report  of  37  percent  for  the  12-month  period  ending 

in  October  2013.  However,  the  program  reported  an 
improved  availability  in  October  2014,  reaching  an  average 
rate  of  51  percent  for  the  fleet  of  90  aircraft  and  breaking 
50  percent  for  the  first  time,  but  still  short  of  the  program 
objective  of  60  percent  set  for  the  end  of  CY14.  The 
bump  in  availability  in  October  brought  the  fleet  12-month 
average  to  39  percent. 

-  Measures  of  reliability  and  maintainability  that  have  ORD 
requirements  have  improved  since  last  year,  but  all  nine 
reliability  measures  (three  for  each  variant)  are  still  below 
program  target  values  for  the  current  stage  of  development. 
The  reliability  metric  that  has  seen  the  most  improvement 
since  May  2013  is  not  an  ORD  requirement,  but  a  contract 
specification  metric,  mean  flight  hour  between  failure 
scored  as  “design  controllable”  (which  are  equipment 
failures  due  to  design  flaws).  For  this  metric,  the  F-35B 
and  F-35C  are  currently  above  program  target  values, 

and  F-35  A  is  slightly  below  the  target  value,  but  has  been 
above  the  target  value  for  several  months  over  the  last  year. 

F-35  Fleet  Availability 

•  Aircraft  availability  is  determined  by  measuring  the  percent 
of  time  individual  aircraft  are  in  an  “available”  status, 
aggregated  over  a  reporting  period  (e.g.,  monthly).  Aircraft 
that  are  not  available  are  assigned  to  one  of  three  categories 
of  status:  Not  Mission  Capable  for  Maintenance  (NMC-M); 
Not  Mission  Capable  for  Supply  (NMC-S);  and  in  depot. 

-  The  program  added  this  third  category  for  tracking  fleet 
status  in  January  2014  as  the  number  of  aircraft  entering 
the  depot  for  modifications  or  receiving  modifications 
or  repair  by  a  depot  field  team  at  the  home  station  began 
to  increase.  Prior  to  January  2014,  these  aircraft  were 
assigned  as  Non-Possessed  (NP)  or  Out  Of  Reporting 
(OOR)  for  depot-level  actions  under  an  NMC-M  status. 

-  The  program  established  new  goals  for  all  three  of  these 
unavailable  statuses  for  2014.  The  NMC-M  goal  is 

15  percent,  NMC-S  is  10  percent,  and  depot  status  is 
15  percent.  These  three  non-available  statuses  sum  to 
40  percent,  supporting  the  program’s  availability  goal 
of  60  percent  for  the  fleet  by  the  end  of  CY14.  The  goal 
of  60  percent  is  an  interim  program  goal  and  does  not 
represent  the  availability  needed  for  combat  operations, 
nor  the  80  percent  needed  to  accomplish  IOT&E  in  an 
operationally  realistic  manner. 

•  Aircraft  monthly  availability  averaged  39  percent  for  the 
12-month  period  ending  October  2014  in  the  training  and 
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operational  fleet,  with  no  statistical  trend  of  improvement 
for  the  first  11  months.  In  October  2014,  availability 
jumped  to  a  reported  5 1  percent  (fleet  size  of  90  aircraft), 
a  12  percent  increase  from  the  previous  month,  and  the 
largest  month-to-month  change  since  March  2013  (fleet 
size  of  27  aircraft).  Month-to-month  variance  in  average 
availability  should  decrease  as  the  fleet  size  increases.  The 
improved  availability  was  seen  at  most  operating  locations, 
and  resulted  from  roughly  equal  improvements  in  the 
NMC-M  and  NMC-S  rates.  Historically  NMC-M  and 
NMC-S  have  tended  to  move  in  opposite  directions;  the 
improvement  in  one  being  negated  by  the  increase  in  the 
other. 

•  Aircraft  availability  rates  by  operating  location  for  the 
12-month  period  ending  October  2014  are  summarized  in 
the  table  below.  The  first  column  indicates  the  average 
availability  achieved  for  the  whole  period,  while  the 
maximum  and  minimum  columns  represent  the  range  of 
monthly  availabilities  reported  over  the  period.  The  number 
of  aircraft  assigned  at  the  end  of  the  reporting  period  is 
shown  as  an  indicator  of  potential  variance  in  the  rates.  Sites 
are  arranged  in  order  of  when  each  site  began  operation  of 
any  variant  of  the  F-35,  and  then  arranged  by  variant  for  sites 
operating  more  than  one  variant. 


F-35  AVAILABILITY  FOR  12-MONTH  PERIOD  ENDING  OCTOBER  20141 

Operational 

Site 

Average 

Maximum 

Minimum 

Aircraft 

Assigned 

Total  Fleet 

39% 

51% 

35% 

O 

CT> 

Eglin  F-35  A 

39% 

55% 

32% 

28 

Eglin  F-35B 

41% 

54% 

25% 

11 

Eglin  F-35C 

50% 

64% 

24% 

10 

Yuma  F-35B 

33% 

49% 

24% 

15 

Edwards  F-35A 

43% 

57% 

19% 

7 

Nellis  F-35A 

28% 

51% 

2% 

4 

Luke  F-35A3 

50% 

58% 

23% 

10 

Beaufort  F-35B4 

37% 

49% 

4% 

4 

1 .  Data  do  not  include  SDD  aircraft. 
2.  Total  includes  1  OT  F-35B  at  Edwards  that  is  not  broken  out  in  table 

3.  Luke  F-35A  data  began  in  April  201 4 
4.  Beaufort  F-35  B  data  began  in  July  2014 

•  Sites  that  show  extreme  maximum  or  minimum  availability 
values  tend  to  have  small  fleet  sizes;  for  example, 

Nellis  AFB,  Nevada,  had  only  four  F-35A  aircraft  for  the 
majority  of  the  reporting  period.  F-35B  operations  began  at 
Edwards  AFB,  California,  in  October,  when  a  single  aircraft 
was  transferred  from  Yuma  MCAS.  Availability  of  that 
aircraft  is  not  broken  out  separately,  but  is  included  in  the 
whole  fleet  calculation. 

•  The  NMC-M  rate  was  relatively  steady  at  an  average  of 
26  percent  for  the  12-month  period  ending  October  2014, 
nearly  twice  the  goal  for  2014,  excluding  the  depot  for  this 
entire  period.  A  substantial  amount  of  NMC-M  down  time 
continues  to  be  the  result  of  field  maintenance  organizations 


waiting  for  technical  dispositions  or  guidance  from  the 
contractor  on  how  to  address  a  maintenance  issue  that  has 
grounded  an  aircraft.  These  Action  Requests  (ARs)  are  a 
result  of  incomplete  or  inadequate  technical  information  in 
the  field,  in  the  fonn  of  Joint  Technical  Data  (JTD).  While 
JTD  validation  has  progressed  (see  separate  section  below), 
the  complexity  of  AR’s  is  increasing,  leading  to  longer  times 
to  receive  final  resolution.  Reducing  the  rate  of  ARs  or 
decreasing  the  response  time  to  the  ARs  will  improve  (lower) 
NMC-M  rates.  High  Mean  Times  To  Repair  (MTTR),  the 
average  maintenance  time  to  fix  a  single  discrepancy,  are 
experienced  in  all  variants.  This  also  contributes  to  the 
persistently  high  NMC-M  rate. 

Over  the  same  12-month  period,  the  NMC-S  rate 
displayed  an  improving  trend,  peaking  at  27  percent  in 
November  2013,  decreasing  to  rates  in  the  high  10s  to 
low  20s  by  mid-2014,  and  reaching  a  minimum  of  15  percent 
in  October.  In  2013,  the  Program  Office  predicted  that  better 
contracting  performance  and  the  maturing  supply  system 
would  result  in  improved  supply  support,  which  would  in 
turn  result  in  lower  NMC-S  rates  by  late  2014.  Although 
the  trend  is  favorable,  the  rate  of  improvement  is  not  yet 
fast  enough  to  allow  the  program  to  achieve  their  goal  of 
10  percent  NMC-S  by  the  end  of  2014.  If  the  current  trend 
continues,  the  program  could  reach  this  target  in  early-  to 
mid-2015. 

A  large  portion  of  the  fleet  began  cyc  ling  through  the  depot 
for  Block  2B  modifications  made  necessary  by  concurrent 
development,  exerting  downward  pressure  on  overall  fleet 
availability.  The  program  began  reporting  the  percentage 
of  the  fleet  in  depot  status  starting  in  January  2014  at 
13  percent.  Since  then,  it  has  risen  to  as  high  as  18  percent 
in  July  2014,  and  was  at  11  percent  by  the  end  of  October. 
Current  plans  show  over  10  percent  of  the  operational 
aircraft  inventory  will  be  in  depot  status  for  Block  2B 
modifications  through  at  least  mid-2015  (either  at  a  dedicated 
facility  or  being  worked  on  by  a  depot  field  team  at  the 
home  station).  If  the  Services  elect  to  upgrade  all  early 
production  aircraft  to  Block  3F  capability,  these  aircraft  will 
again  be  scheduled  for  depot-level  modifications  (operational 
test  aircraft  must  be  modified.)  All  necessary  depot-level 
modifications  are  not  yet  identified  for  Block  3F,  as  testing 
and  development  are  not  complete.  Therefore,  the  impact 
on  availability  due  to  Block  3F  modifications  in  the  2016 
through  2018  timeframe  is  unknown. 

Although  depot  modifications  reduce  overall  fleet 
availability,  they  potentially  improve  availability  once 
the  aircraft  is  out  of  depot  by  replacing  low  reliability 
components  with  improved  versions,  such  as  the  270  Volt 
Battery  Charger  and  Control  Unit.  Any  increased  availability 
from  reliability  improvements  will  take  time  to  manifest  in 
the  fleet  wide  metrics,  not  showing  more  strongly  until  the 
majority  of  aircraft  have  been  modified. 

Low  availability  rates,  in  part  due  to  poor  reliability,  are 
preventing  the  fleet  of  fielded  operational  F-35  aircraft  (all 
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variants)  from  achieving  planned,  Service-funded  flying 
hour  goals.  Original  Service  bed-down  plans  were  based  on 
F-35  squadrons  ramping  up  to  a  steady  state,  fixed  number  of 
flight  hours  per  tail  per  month,  allowing  for  the  projection  of 
total  fleet  flight  hours. 

-  In  November  2013,  a  new  “modelled  achievable”  flight 
hour  projection  was  created  since  low  availability  was 
preventing  the  full  use  of  bed-down  plan  flight  hours.  The 
revised  model  accounted  for  some  actual  fleet  maintenance 
and  supply  data,  and  made  assumptions  about  many 
factors  affecting  availability  in  the  coming  years  to  predict 
the  number  of  flight  hours  the  fleet  could  generate  in  future 
months. 

-  Through  October  30,  2014,  the  fleet  had  flown 
approximately  72  percent  of  the  modelled  achievable 
hours  because  availability  had  not  increased  in  accordance 
with  assumptions.  Planned  versus  achieved  flight  hours, 
for  both  the  original  plans  and  the  modelled  achievable, 
till'd  ugh  October  30,  2014,  by  variant,  for  the  fielded 
production  aircraft  are  shown  in  the  table  below. 


F-35  FLEET  PLANNED  VS.  ACHIEVED  FLIGHT  HOURS  AS  OF  OCTOBER  30, 2014 

Variant 

Original  Bed-Down  Plan 
Cumulative  Flight  Hours 

"Modelled  Achievable" 
Cumulative  Flight  Hours 

Estimated 

Planned 

Achieved 

Percent 

Planned 

Estimated 

Planned 

Achieved 

Percent 

Planned 

F-35A 

11,500 

6,347 

55% 

9,000 

6,347 

71% 

F-35B 

8,500 

6,085 

72% 

7,500 

6,085 

81% 

F-35C 

1,800 

910 

51% 

1,600 

910 

57% 

Total 

21,800 

13,342 

61% 

18,600 

13,342 

72% 

F-35  Fleet  Reliability 

•  Aircraft  reliability  is  assessed  using  a  variety  of  metrics,  each 
characterizing  a  unique  aspect  of  overall  weapon  system 
reliability. 

-  Mean  Flight  Flours  Between  Critical  Failure  (MFF1BCF) 
includes  all  failures  that  render  the  aircraft  not  safe  to 
fly,  and  any  equipment  failures  that  would  prevent  the 
completion  of  a  defined  F-35  mission.  It  includes  failures 
discovered  in  the  air  and  on  the  ground. 

-  Mean  Flight  Flours  Between  Removal  (MFF1BR)  gives 
an  indication  of  the  degree  of  necessary  logistical  support 
and  is  frequently  used  in  determining  associated  costs. 

It  inc  ludes  any  removal  of  an  item  from  the  aircraft  for 
replacement  with  a  new  item  from  the  supply  chain.  Not 
all  removals  are  failures,  and  some  failures  can  be  fixed 
on  the  aircraft  without  a  removal.  For  example,  some 
removed  items  are  later  determined  to  have  not  failed 
when  tested  at  the  repair  site.  Other  components  can  be 
removed  due  to  excessive  signs  of  wear  before  a  failure, 
such  as  worn  tires. 

-  Mean  Flight  Flours  Between  Maintenance  Event, 
unscheduled  (MFF1BME)  is  useful  primarily  for 
evaluating  maintenance  workload.  It  includes  all  failures, 
whether  inherent  or  induced  by  maintenance  actions,  that 


led  to  maintenance  and  all  unscheduled  inspections  and 
servicing  actions. 

-  Mean  Flight  Flours  Between  Failure,  Design  Controllable 
(MFHBFDC)  includes  failures  of  components  due  to 
design  flaws  under  the  purview  of  the  contractor,  such 
as  the  inability  to  withstand  loads  encountered  in  normal 
operation.  Failures  of  Government  Furnished  Equipment 
(GFE)  and  failures  induced  by  improper  maintenance 
practices  are  not  included. 

The  F-35  program  developed  reliability  growth  projections 
for  each  variant  throughout  the  development  period  as  a 
function  of  accumulated  flight  hours.  These  projections  are 
shown  as  growth  curves,  and  were  established  to  compare 
observed  reliability  with  target  numbers  to  meet  the  threshold 
requirement  at  maturity,  defined  by  75,000  flight  hours  for  the 
F-35A  and  F-35B,  and  by  50,000  flight  hours  for  the  F-35C, 
and  200,000  cumulative  fleet  flight  hours.  In  November  2013, 
the  program  discontinued  reporting  against  these  curves  for 
all  ORD  reliability  metrics,  and  retained  only  the  curve  for 
MFHBF  DC,  which  is  the  only  reliability  metric  included  in 
the  JSF  Contract  Specification  (JCS).  The  growth  curves  for 
the  other  metrics  have  been  re-constructed  analytically  and  are 
used  in  the  tables  below  for  comparison  to  achieved  values, 
but  are  not  provided  by  the  Program  Office. 

As  of  October  2014,  the  F-35,  including  operational  and  flight 
test  aircraft,  had  accumulated  approximately  22,000  flight 
hours,  or  slightly  more  than  1 1  percent  of  the  total 
200,000-hour  maturity  mark  defined  in  the  ORD. 

Since  May  2013,  the  program  has  reported  Reliability  and 
Maintainability  (R&M)  metrics  on  a  three-month  rolling 
window  basis  meaning,  for  example,  the  MFHBCF  rate 
published  for  a  month  accounts  only  for  the  critical  failures 
and  flight  hours  of  that  month  and  the  two  previous  months. 
Before  May  2013,  R&M  metrics  were  reported  on  a 
cumulative  basis.  The  switch  to  a  three-month  rolling  window 
is  intended  to  give  a  more  accurate  account  of  current,  more 
production-representative  aircraft  performance,  and  eliminate 
the  effect  of  early  history  when  the  SDD  aircraft  were  very 
immature;  however,  this  process  can  create  significant 
month-to-month  variability  in  reported  numbers. 

A  comparison  of  current  observed  and  projected  interim  goal 
MFHBCF  rates,  with  associated  flight  hours,  is  shown  in  the 
first  table  on  the  following  page.  Threshold  at  maturity  and 
the  values  in  the  FY13  DOT&E  report  are  shown  for  reference 
as  well. 

Similar  tables  comparing  current  observed  and  projected 
interim  goals  for  MFHBR,  MFHBME,  and  MFHBF  DC  rates 
for  all  three  variants  are  also  provided.  MFHBF  DC  is  a 
contract  specification,  and  its  JCS  requirement  value  is  shown 
in  lieu  of  an  ORD  threshold. 

The  large  number  of  flight  hours  and  events  in  each 
three-month  rolling  window  supporting  the  observed 
reliability  metrics  in  the  tables  above  provide  statistical 
evidence  that  the  program  experienced  reliability  growth  in 
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F-35  RELIABILITY:  MFHBR  (HOURS) 


Variant 

ORD  Threshold 

Values  as  of  August  31 , 201 4 

Values  as  of  August  201 3 

Flight 

Hours 

MFHBR 

Cumulative 

Flight 

Hours 

Interim  Goal 
to  Meet  ORD 
Threshold 
MFHBR 

Observed 
MFHBR 
(3  Mos.  Rolling 
Window) 

Observed 
Value  as 
Percent  of 
Goal 

Cumulative 

Flight 

Hours 

Observed 
MFHBR 
(3  Mos.  Rolling 
Window) 

F-35A 

75,000 

6.5 

8,834 

4.8 

3.1 

65% 

4,204 

2.5 

F-35B 

75,000 

6.0 

7,039 

4.3 

2.5 

58% 

3,286 

1.4 

F-35C 

50,000 

6.0 

2,046 

3.9 

2.3 

59% 

903 

1.6 

F-35  RELIABILITY:  MFHBME  (HOURS) 


Variant 

ORD  Threshold 

Values  as  of  August  31 , 201 4 

Values  as  of  August  201 3 

Flight 

Hours 

MFHBME 

Cumulative 

Flight 

Hours 

Interim  Goal 
to  Meet  ORD 
Threshold 
MFHBME 

Observed 
MFHBME 
(3  Mos.  Rolling 
Window) 

Observed 
Value  as 
Percent  of 
Goal 

Cumulative 

Flight 

Hours 

Observed 
MFHBME 
(3  Mos.  Rolling 
Window) 

F-35A 

75,000 

2.0 

8,834 

1.5 

0.85 

57% 

4,204 

0.78 

F-35B 

75,000 

1.5 

7,039 

1.1 

0.96 

87% 

3,286 

0.46 

F-35C 

50,000 

1.5 

2,046 

0.9 

0.84 

93% 

903 

0.35 

F-35  RELIABILITY:  MFHBF„DC  (HOURS) 


Variant 

JCS  Requirement 

Values  as  of  August  31, 2014 

Values  as  of  August  201 3 

Flight 

Hours 

MFHBF_ 

DC 

Cumulative 

Flight 

Hours 

Interim  Goal 
to  Meet  JCS 
Requirement 
MFHBFJX 

Observed 
MFHBFJX 
(3  Mos.  Rolling 
Window) 

Observed 
Value  as 
Percent  of 
Goal 

Cumulative 

Flight 

Hours 

Observed 
MFHBFJX 
(3  Mos.  Rolling 
Window) 

F-35A 

75,000 

6.0 

8,834 

4.2 

4.0 

95% 

4,204 

2.8 

F-35B 

75,000 

4.0 

7,039 

2.7 

3.5 

1 30% 

3,286 

1.9 

F-35C 

50,000 

4.0 

2,046 

2.4 

3.6 

1 50% 

903 

1.5 

all  metrics  and  all  variants 
between  August  2013  and 
August  2014. 

The  critical  failure  rates  for 
all  three  variants  were  below 
threshold  values  and  below 
projected  interim  goals. 

Due  to  the  large  variability 
in  month-to-month  reported 
values,  however,  the  high 
apparent  growth  for  both 
the  F-35B  and  F-35C  from 
the  data  point  values  above 
may  not  be  characteristic 
of  the  actual  growth,  with 
August  2013  being  notably 
below  average  for  those 
variants,  and  August  2014 
being  substantially  above 
average. 

All  variants  are  below 
their  threshold  values  and 
projected  interim  goals  for 
MFHBR  and  MFHBME. 

Design  controllable  failure 
rate  is  the  only  metric  where 
any  variants  exceed  the 
interim  goal;  as  shown  in 
the  table  with  the  F-35B 
and  F-35C.  For  all  variants, 
the  degree  of  improvement 
in  MFHBFDC  by 
August  2014,  relative  to 
the  May  2013  value,  is 
greater  than  the  degree  of 
improvement  for  all  other 
reliability  metrics.  This 
indicates  the  improvement 
in  the  contract  specification 
metric  of  MFHBF  DC  is 
not  translating  into  equally 
large  improvement  in  the 
other  reliability  metrics,  which  are  operational  requirements. 
DOT&E  conducted  an  in-depth  study  of  reliability  growth 
in  MFHBR  and  in  MFHBME  for  the  period  from  July  2012 
through  October  2013.  Reliability  growth  was  modeled 
using  the  Duane  Postulate,  which  characterizes  growth  by  a 
single  parametric  growth  rate.  Mathematically,  the  Duane 
Postulate  assesses  growth  rate  as  the  slope  of  the  best  fit  line 
when  the  natural  logarithm  of  the  cumulative  failure  rate 
is  plotted  against  the  natural  logarithm  of  cumulative  flight 
hours.  A  growth  rate  of  zero  would  indicate  no  growth,  and 
a  growth  rate  of  1.0  is  the  theoretical  upper  limit,  indicating 
instantaneous  growth  from  a  system  that  exhibits  some 


failures  to  a  system  that  never  fails.  The  closer  the  growth 
rate  is  to  1.0  the  faster  the  growth,  but  the  relationship 
between  assessed  growth  rates  is  not  linear,  due  to  the 
logarithmic  nature  of  the  plot.  For  example  a  growth  rate 
of  0.4  would  indicate  reliability  growth  much  higher  than 
twice  as  fast  as  a  growth  rate  of  0.2. 

-  Only  the  F-35A  and  F-35B  variants  were  investigated  due 
to  a  low  number  of  flight  hours  on  the  F-35C.  The  study 
evaluated  the  current  growth  rate,  then,  using  that  rate, 
projected  the  reliability  metric  to  the  value  expected  at 
maturity. 
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The  study  also  evaluated  the  growth  rate  needed  to  meet 
the  ORD  threshold  value  at  maturity  (75,000  hours  each 
for  the  F-35A  and  F-35B)  from  the  current  observed 
value  of  the  reliability  metric.  The  results  of  the  study  are 
summarized  in  the  following  table. 


-  For  most  of  the  measures,  the  F-35  must  achieve  a  much 
faster  growth  rate  than  currently  exhibited  in  order  to  meet 
ORD  requirements  by  maturity.  Reliability  growth  rates 
are  very  sensitive  when  calculated  early  in  a  program,  with 
only  relatively  low  numbers  of  flight  hours  (i.e.,  less  than 
10,000),  and  can  differ  significantly  either  on  the  up  or 
down  side  from  growth  rates  calculated  once  a  program  is 
more  mature. 

-  The  above  growth  rates 
were  calculated  with 
around  4,700  flight 
hours  for  the  F-35  A,  and 
3,800  for  the  F-35B.  For 
comparison,  observed 
MFFIBME  growth  rates 
for  several  historical 
aircraft  are  shown  in  the 
table  to  the  right. 

-  The  growth  rates  for  the 
F-35  to  comply  with  ORD  performance  by  maturity  have 
been  demonstrated  in  the  past,  but  only  on  different  type 
aircraft  and  not  on  fighters. 

The  most  recent  90-day  rolling  averages  for  MFHBFDC 

show  more  growth  in  this  metric  than  for  any  other  reliability 

metric  for  the  period  from  May  2013  through  August  2014. 

The  following  contributed  to  the  reported  growth  in 

MFHBFDC. 

-  In  June  2013,  the  program  re-categorized  nut  plate  failures, 
one  of  the  most  common  failures  in  the  aircraft,  as  induced 
failures  rather  than  inherent  failures,  removing  them  from 
the  calculation  of  MFHBF  DC.  Nut  plates  are  bonded 

to  an  aircraft  structure  and  receive  bolt-type  fasteners  to 
hold  removable  surface  panels  in  place.  One  way  nut 
plates  can  fail,  for  example,  is  when  torquing  a  bolt  down 
while  replacing  a  removed  panel,  the  nut  plate  dis-bonds 
from  the  aircraft  structure,  preventing  securing  the  surface 
panel. 

-  Distinguishing  between  inherent  design  failures  and 
induced  failures  can  be  subjective  in  certain  cases.  For 
example,  if  a  maintainer  working  on  the  aircraft  bumps  a 
good  component  with  a  tool  and  breaks  it  while  working 


on  a  different  part  nearby,  it  is  a  judgment  call  whether 
that  is  an  inherent  design  failure  because  the  component 
could  not  withstand  “normal”  wear  and  tear  in 
operational  service,  or  if  it’s  an  induced  failure  because 
the  maintainer  was  “too  rough.” 

Analysis  on  F-35Adata  including  SDD 
and  LR1P  aircraft  from  September  2012  to 
April  2014  shows  a  generally  increasing 
number  of  failures  categorized  as  induced  each 
month  over  the  entire  period,  but  a  generally 
decreasing  number  of  failures  categorized  as 
inherent  for  each  month  since  April  2013.  The 
decreasing  inherent  failure  count  per  month  is 
notable,  as  during  this  period,  the  F-35  A  fleet 
size  and  total  hours  flown  per  month  were 
increasing  steadily. 

Some  of  this  is  due  to  re-categorizing  nut-plate  failures. 
Actual  reliability  growth  can  also  explain  some  of  this, 
as  could  poor  training  leading  to  bad  troubleshooting 
and  maintenance  practices.  Some  of  this  could  also 
be  due  to  re-categorizing  failures  previously  scored 
as  inherent  failures  as  induced  failures.  For  example, 
Program  Office  maintenance  data  records  showed  that 
there  were  twice  as  many  inherent  failures  as  induced 
failures  in  September  2012,  and  there  were  many  more 
inherent  failures  than  induced  for  every  subsequent 
month  through  May  2013.  Then  in  June  2013,  records 
showed  that  there  were  more  than  twice  as  many 
induced  failures  than  inherent  failures,  and  induced 
failures  have  always  been  much  greater  than  inherent 
failures  for  each  month  afterward.  This  sudden  and 
abrupt  reversal  of  the  relationship  between  induced  and 
inherent  failures  across  the  entire  F-35A  fleet  suggests 
that  scoring  failures  differently  (induced  vice  inherent) 
may  result  in  an  increase  in  the  design-controllable 
metric  that  is  not  manifested  in  other  reliability 
metrics. 

Due  to  poorer  than  expected  initial  reliability  of  many 
components,  the  program  has  started  to  re-design  and 
introduce  new,  improved  versions  of  these  parts.  Once  a 
new  version  of  a  component  is  designed,  it  is  considered 
the  production-representative  version.  However,  failed 
components  may  still  be  replaced  by  the  old  version  of 
the  component  in  order  to  keep  aircraft  flying  until  the 
new  version  is  produced  in  enough  quantity  to  proliferate 
to  100  percent  of  the  fleet  and  supply  stock.  During  this 
transition  period,  only  failures  of  the  new  version  of 
the  component  are  counted  as  relevant  to  the  reliability 
metrics,  because  the  old  version  is  no  longer  considered 
production-representative. 

■  This  creates  a  situation  where  not  all  failures  are 

counted  in  the  calculation  of  mean  flight  hours  between 
reliability  events,  but  all  flight  hours  are  counted, 
and  hence  component  and  aircraft  reliability  are 
reported  higher  than  if  all  of  the  failures  were  counted. 
The  result  is  an  increased  estimation  of  reliability 


Metric 

Variant 

October 

2013 

Value 

Current 
Growth  Rate 
from  Duane 
Postulate 

Projected 
Value  at 
75,000 

FH 

ORD 

Threshold 

Projected 
Value  as 
%  ORD 
Threshold 

Growth 

Rate 

Needed  to 
Meet  ORD 

MFHBR 

F-35A 

3.30 

0.129 

4.19 

6.5 

65% 

0.232 

F-35B 

1.87 

0.210 

4.05 

6.0 

68% 

0.305 

MFHBME 

F-35A 

0.82 

0.162 

1.45 

2.0 

73% 

0.241 

F-35B 

0.64 

0.347 

1.74 

1.5 

116% 

0.312 

Aircraft 

MFHBME 
Growth  Rate 

F-1 5 

0.14 

F-16 

0.14 

F-22  (at  35,000  flight  hours) 

0.22 

B-1 

0.13 

"Early"  B-2  (at  5,000  flight  hours) 

0.24 

"Late"  B-2 

0.13 

C-1 7  (at  1 5,000  flight  hours) 

0.35 
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compared  to  an  estimate  using  all  failures,  and  is  highest 
at  the  beginning  of  the  transition  period,  especially  if  the 
initial  batch  of  re-designed  components  is  small. 

■  For  example,  as  of  September  2014,  an  improved 
horizontal  tail  actuator  component  had  been  introduced 
and  installed  on  roughly  30  aircraft  out  of  a  fleet  of 
nearly  100.  Failures  of  the  older  component  were 
not  being  counted  in  the  metrics  at  all  anymore,  but 
flight  hours  from  all  100  aircraft  were  counted.  This 
calculation  could  result  in  the  reported  reliability  of 
that  component  being  increased  by  up  to  a  factor  of 
three  compared  to  reliability  if  all  of  the  horizontal  tail 
actuator  failures  were  counted.  There  are  hundreds  of 
components  on  the  aircraft,  so  a  single  component’s 
increased  estimate  of  reliability  may  have  little  influence 
on  overall  observed  aircraft  reliability.  However,  since 
multiple  components  are  being  upgraded  simultaneously 
due  to  the  unprecedented  and  highly  concurrent  nature 
of  the  F-35  program,  the  cumulative  effect  on  the  overall 
observed  aircraft  reliability  of  the  increased  estimate 
of  reliability  from  all  of  these  components  may  be 
significant. 

-  Tire  assemblies  on  all  F-35  variants  do  not  last  as  long  as 
expected  and  require  very  frequent  replacement.  However, 
only  when  a  tire  failure  is  experienced  on  landing  is  it 
counted  as  a  design  controllable  failure.  The  vast  majority 
of  tires  are  replaced  when  worn  beyond  limits,  and  in 
these  cases  they  are  scored  as  a  “no-defect.”  Thus,  most 
tire  replacements  show  up  in  the  MFHBR  and  MFHBME 
metrics,  but  not  in  MFHBFDC  or  MFHBCF,  even 
though  the  aircraft  is  down  for  unsafe  tires.  The  program 
is  seeking  redesigned  tires  for  all  variants  to  reduce 
maintenance  down  time  for  tire  replacements. 

•  A  number  of  components  have  demonstrated  reliability  much 
lower  than  predicted  by  engineering  analysis,  which  has 
driven  down  the  overall  system  reliability  and/or  led  to  long 
wait  times  for  re-supply.  High  driver  components  affecting 
low  availability  and  reliability  include  the  following, 
grouped  by  components  common  to  all  variants  followed  by 
components  failing  more  frequently  on  a  particular  variant  or 
completely  unique  to  it,  as  shown  below: 


HIGH  DRIVER  COMPONENTS  AFFECTING  LOW  AVAILABILITY  AND 
RELIABILITY 

Common  to  All  Variants 

Additional  High  Drivers 
by  Variant 

F-35A 

•  Avionics  Processors 

•  Main  Landing  GearTires 

•  Thermal  Management  System 

•  Ejection  Seat  Assembly 

•  Panoramic  Cockpit  Display 
Electronics  Unit 

•  Low  Observable  Cure  Parameters 

•  Helmet  Display  Unit 

•  Seat  Survival  Kit 

•  Igniter-Spark,  Turbine  Engine 

•  On-Board  Oxygen  Generating 
System 

•  Exhaust  Nozzle  Assembly 

•  Exhaust  Nozzle  Converging- 
Diverging  Link 

F-35B 

•  Upper  Lift  Fan  Door  Actuator1 

•  270  Volt  DC  Battery 

F-35C 

•  Data  Transfer  Cartridge 

•  Solenoid  Operated  Fuel  Valve 

1.  Unique  to  the  F-35B 

Maintainability 

•  The  amount  of  time  spent  on  maintenance  for  all  variants 
exceeds  that  required  for  mature  aircraft.  Two  measures 
used  to  gauge  this  time  are  Mean  Corrective  Maintenance 
Time  for  Critical  Failures  (MCMTCF)  and  Mean  Time  To 
Repair  (MTTR)  for  all  unscheduled  maintenance.  MCMTCF 
measures  active  maintenance  time  to  correct  only  the 
subset  of  failures  that  prevent  the  JSF  from  being  able  to 
perform  a  defined  mission,  and  indicates  how  long  it  takes, 
on  average,  to  return  an  aircraft  to  mission  capable  status. 
MTTR  measures  the  average  active  maintenance  time  for  all 
unscheduled  maintenance  actions,  and  is  a  general  indicator 
of  ease  and  timeliness  of  repair.  Both  measures  include 
active  touch  labor  time  and  cure  times  for  coatings,  sealants, 
paints,  etc.,  but  do  not  include  logistics  delay  times  such  as 
how  long  it  takes  to  receive  shipment  of  a  replacement  part. 

•  The  tables  below  compare  measured  MCMTCF  and  MTTR 
values  for  the  three-month  period  ending  August  2014  to 
the  ORD  threshold  and  the  percentage  of  the  value  to  the 
threshold  for  all  three  variants.  The  tables  also  show  the 
value  reported  in  the  FY13  DOT&E  Annual  Report  for 
reference.  For  the  F-35A  and  F-35C,  MCMTCF  increased 
(worsened)  over  the  last  year  while  MCMTCF  for  the  F-35B 
showed  slight  improvement.  For  all  variants,  MTTR  showed 
improvement  over  the  last  year.  Both  maintainability 
measures  for  all  variants  are  well  above  (worse  than)  the 
ORD  threshold  value  required  at  maturity. 


F-35  MAINTAINABILITY:  MCMTCF  (HOURS) 

Variant 

ORD 

Threshold 

Values  as  of 
August  31, 2014 
(3  Mos.  Rolling 
Window) 

Observed 
Value  as 
Percent  of 
Threshold 

Values  as  of 
August  201 3 
(3  Mos.  Rolling 
Window) 

F-35A 

4.0 

15.6 

390% 

12.1 

F-35B 

4.5 

15.2 

338% 

15.5 

F-35C 

4.0 

11.2 

280% 

9.6 

F-35  MAINTAINABILITY:  MTTR  (HOURS) 

Variant 

ORD 

Threshold 

Values  as  of 
August  31, 2014 
(3  Mos.  Rolling 
Window) 

Observed 
Value  as 
Percent  of 
Threshold 

Values  as  of 
August  201 3 
(3  Mos.  Rolling 
Window) 

F-35A 

2.5 

8.6 

344% 

9.2 

F-35B 

3.0 

7.5 

250% 

8.9 

F-35C 

2.5 

6.6 

264% 

7.7 

•  More  in  depth  trend  analysis  between  May  2013  and 
August  2014  shows  that  the  MTTR  for  the  F-35  A  and  F-35C 
variants  have  been  decreasing  slowly,  while  the  MTTR  for 
the  F-35B  has  been  growing  slightly,  with  all  exhibiting 
high  month-to-month  variability.  Over  the  same  period,  the 
MCMTCF  values  for  the  F-35B  and  F-35C  were  increasing 
slightly  and  flat  for  the  F-35A,  but  again  with  very  high 
monthly  variability. 

•  Several  factors  likely  contribute  to  extensive  maintenance 
time,  especially  long  cure  times  for  Low  Observable  repair 
materials.  The  Program  Office  is  addressing  this  issue  with 
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new  materials  that  can  cure  in  12  hours  vice  48  for  example, 
but  some  of  these  materials  may  require  freezer  storage, 
making  re-supply  and  shelf  life  verification  in  the  field  or  at 
an  austere  operating  location  more  difficult. 

-  The  immaturity  of  the  system  overall,  including  training 
system  immaturity,  lack  of  maintainer  experience  on  such 
a  new  aircraft,  and  incompletely  written  and  verified,  or 
poorly  written,  JTD  may  all  also  contribute  to  protracted 
maintenance  times. 

-  Additionally,  design  factors  of  the  aircraft  itself  make 
affecting  certain  repairs  difficult  and  time-consuming. 

Field  maintainers  have  reported  poor  cable  routing  behind 
panels  that  interferes  with  required  maintenance,  and 
awkward  placement  of  some  components,  which  makes 
removing  and  replacing  them  slow,  and  increases  the 
chances  they  will  induce  a  failure  in  a  nearby  component 
working  with  tools  in  confined  spaces. 

-  Scoring  also  affects  higher  than  expected  MTTR  values. 
Discrepancies  for  which  maintainers  have  to  attempt 
multiple  solutions  before  finding  a  true  fix  are  being 
re-scored  as  a  single  event,  while  in  the  past  they  were 
documented  as  multiple  repair  attempts,  each  with  its 
own  MTTR.  The  individual  MTTRs  for  these  attempted 
repairs  are  now  rolled  up  into  the  single,  re-scored  event. 
Improved  diagnostics  and  training  can  reduce  MTTR  by 
pointing  maintainers  to  the  true  root  cause  of  discrepancies 
more  quickly. 

Autonomic  Logistics  Information  System  (ALIS) 

•  The  program  develops  and  fields  ALIS  in  increments 
similar  to  the  mission  systems  capability  in  the  air  vehicle. 
Overall,  ALIS  is  behind  schedule,  has  several  capabilities 
delayed  or  deferred  to  later  builds,  and  has  been  fielded 
with  deficiencies.  The  program  does  not  have  a  dedicated 
end-to-end  developmental  testing  venue  for  ALIS  and  has 
relied  on  feedback  from  the  field  locations  for  identifying 
deficiencies.  Though  some  of  the  early  deficiencies  have 
been  addressed,  ALIS  continues  to  be  cumbersome  to  use 
and  inefficient,  and  requires  the  use  of  workarounds  for 
deficiencies  awaiting  correction.  The  program  has  tested 
ALIS  software  versions  at  the  Edwards  flight  test  center, 
including  a  formal  Logistics  Test  and  Evaluation  (LT&E) 
of  ALIS  software  versions  1.0.3  and  2.0.0.  These  formal 
test  periods  had  limitations,  however,  as  the  ALIS  that 
supports  the  developmental  test  aircraft  is  different  than 
the  production  ALIS  hardware  at  fielded  units.  As  a  result, 
the  program  has  begun  limited  testing  of  software  updates 
at  fielded  operational  sites  and  will  expand  this  testing  in 
CY15.  The  program  should  ensure  adequate  testing  of  ALIS 
software  upgrades  on  operationally-representative  hardware 
is  complete  prior  to  fielding  to  operational  units. 

-  In  the  last  year,  the  Program  Office  adjusted  the  schedule 
and  incremental  development  plans  for  ALIS  hardware 
and  software  capability  releases  three  times.  These 
adjustments  were  necessary  to  align  ALIS  capabilities  with 


Service  requirements  to  support  planned  IOC  declaration 
dates. 

■  In  December  2013,  the  program  re -planned  the 
schedule  and  capability  release  of  ALIS  2.0.0,  the  next 
version  to  be  fielded,  moving  the  initial  release  from 
November  2014  to  January  2015. 

■  In  February  2014,  the  program  adjusted  the  schedule 
and  release  plans  for  the  follow-on  version  of  ALIS, 
version  2.0.1.  The  schedule  for  fielding  was  adjusted 
by  three  months  (from  March  2015  to  June  2015)  and 
the  life  limited  parts  management  (LLPM)  module 
was  deferred  to  later  increments  of  ALIS.  Because 
of  delays  in  development,  the  LLPM  capability  was 
split  into  two  increments  (initial  and  final);  the  initial 
increment  will  be  fielded  with  ALIS  2.0.2  and  aligned 
to  support  Air  Force  IOC  plans,  and  the  final  increment 
of  LLPM  will  be  fielded  in  ALIS  2.0.3. 

■  In  November  2014,  the  program  adjusted  the 
schedule  and  release  plans  again,  moving  the  final 
increment  of  the  LLPM  to  ALIS  3.0.0  and  accelerating 
the  integration  of  an  upgraded  processor  from 

ALIS  3.0.0  to  ALIS  2.0.2,  eliminating  the  need  for 
ALIS  release  2.0.3.  The  content  previously  planned 
for  ALIS  3.0.0  will  be  renamed  3.0.1.  The  program’s 
planned  release  dates  are  July  2017  for  ALIS  3.0.0  and 
July  2018  for  ALIS  3.0.1. 

■  A  Windows  server  update  has  moved  forward  to  an 
earlier  ALIS  release,  from  ALIS  3.0.0  to  2.0.1,  which 
the  program  plans  to  field  in  June  2015. 

•  During  CY14,  the  program  accomplished  the  following 
with  ALIS  software  development  and  fielding: 

-  The  program  completed  the  migration  of  operational 
units  from  older  versions  to  ALIS  1.0.3  (the  current 
fielded  version)  in  January  2014  as  planned, 
followed  by  an  updated  version  in  February  2014 
(version  1.0.3A3.3.1),  which  included  limited  fixes  for 
deficiencies  identified  during  testing  in  late  CY12  and 
early  CY13.  ALIS  1.0. 3A. 3. 3.1  has  reduced  screen 
refresh  and  load  times  compared  to  1.0.3,  and  reduced 
the  number  of  nuisance/false  health  reporting  codes; 
however,  time-consuming  workarounds  are  required 
to  determine  and  update  the  readiness  of  aircraft  to  fly 
missions.  The  following  are  examples  of  workarounds. 

■  Additional  steps  required  to  process  aircraft  health 
information  to  be  compatible  with  the  Exceedance 
Management  System,  which  is  not  integrated  into 
ALIS. 

■  Manual  entry  of  information  into  ALIS  to  track 
consumables  such  as  oil  usage. 

■  Frequent  submission  of  formal  ARs  to  Lockheed 
Martin  for  assistance,  because  troubleshooting 
functionality  is  incomplete. 

■  Manual  correlation  of  health  reporting  codes  between 
ALIS  domains. 
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-  In  future  versions  of  ALIS,  the  program  plans  to 
address  the  above  workarounds  and  include  three  key 
requirements  identified  by  the  Services  as  needed  for 
IOC: 

■  Integration  with  a  new  deployable  ALIS  standard 
operating  unit  (SOU)  hardware  (SOU  V2,  described 
below) 

■  Support  of  detached,  sub-squadron  operations  at 
deployment  locations  away  from  the  main  operating 
base 

■  Distributed  maintenance  operations  allowing 
supervisors  to  verify  completion  of  maintenance 
operations  from  various  locations  at  the  main  or 
deployed  operating  base  (i.e.,  dynamic  routing). 

-  The  next  major  increment  of  ALIS  software, 
version  2.0.0,  began  testing  with  the  mission 
systems  developmental  test  aircraft  at  Edwards  AFB 
in  September  2014.  The  program  plans  to  field 
version  2.0.0  starting  in  January  2015.  The  ALIS  2.0.0 
upgrade  includes  integrated  exceedance  management, 
Windows  7,  recording  of  structural  health  data  for  use  in 
the  future  development  of  prognostic  health  capabilities, 
and  continued  optimization  efforts  with  improvements  to 
data  structures  and  database  tuning. 

■  Testing  of  the  screen  refresh  times  for  ALIS  2.0.0  in 
a  laboratory  environment  has  shown  improvement 
compared  to  those  observed  with  ALIS  1.0.3A3.3.1. 

For  example,  in  a  simulated  environment  supporting 
28  aircraft,  squadron  health  management  debrief  time 
decreased  from  101  seconds  to  less  than  5  seconds  after 
implementation  of  several  cycles  of  improvements. 
Actual  fielded  performance  is  unknown. 

■  Preliminary  results  from  the  LT&E  of  ALIS  2.0.0  show 
that  multiple  deficiencies  from  past  evaluations  remain 
unresolved,  and  the  system  demonstrated  deficiencies 
in  new  capabilities.  Although  results  have  not  been 
finalized  with  a  deficiency  review  board,  the  initial 
LT&E  report  indicates: 

»  A  critical  deficiency  noted  in  the  LT&E  of 
ALIS  1.0.3  for  the  failure  of  the  manual  control 
override  to  work  correctly,  which  results  in  the 
incorrect  reporting  of  the  air  vehicle  status  as  not 
mission  capable  in  the  squadron  health  management 
function  of  ALIS,  has  not  been  corrected  in  ALIS 
2.0.0. 

»  ALIS  2.0.0  demonstrated  4  additional  critical 
deficiencies  and  53  serious  deficiencies. 

»  Exceedance  management  has  been  integrated  into 
ALIS  2.0.0  but  exhibited  processing  delays. 

»  The  test  site  was  unable  to  complete  testing  of  all 
ALIS  2.0.0  functionality  because  the  site  lacks 
a  squadron  operating  unit  and  instead  relied  on 
data  transfers  between  Edwards  AFB  and  Fort 
Worth,  Texas.  The  test  team  recommended  that  the 
remaining  tests  be  conducted  at  an  operating  location 
with  representative  hardware. 


-  ALIS  2.0.0  will  provide  the  basis  for  incremental  builds 
(versions  2.0. 1  and  2.0.2),  which  are  intended  to  be  fielded 
in  support  of  Marine  Corps  IOC  and  the  Air  Force  IOC 
declarations,  respectively. 

■  The  program  plans  to  deliver  ALIS  2.0. 1  to  the  flight 
test  center  in  February  2015,  conduct  a  formal  LT&E, 
in  preparation  for  fielding  in  July  2015,  which  is 

the  current  objective  date  for  Marine  Corps  IOC. 

ALIS  2.0.1  software  will  align  with  a  new  hardware 
release  (SOU  version  2)  that  will  improve  deployability 
and  will  include  fault  isolation  improvements  and  a 
Windows  server  update. 

■  To  support  the  Marine  Corps  preparation  for  IOC,  the 
program  plans  to  release  ALIS  2.0.1  in  May  2015  to 
Yuma  MCAS,  Arizona,  simultaneous  with  the  planned 
delivery  of  the  deployable  ALIS  hardware  system  for 
limited  validation  and  verification  testing  of  the  software 
prior  to  release  to  the  rest  of  the  fielded  units.  Though 
the  current  ALIS  release  schedule  leaves  no  margin  for 
delay  to  meeting  the  Marine  Corps  IOC  objective  date 
in  July,  fielding  ALIS  2.0. 1  before  formal  testing  and  fix 
verification  is  complete  may  result  in  the  continued  need 
for  workarounds  to  support  field  operations. 

-  The  program  has  scheduled  ALIS  2.0.2  fielding,  which 
is  required  to  meet  Air  Force  IOC  requirements,  for 
December  2015.  It  will  provide  a  sub-squadron  reporting 
capability  that  allows  air  vehicle  status  reporting  of 
deployed  assets  back  to  the  parent  SOU,  and  adds  dynamic 
routing,  which  allows  delivery  of  messages  and  data  via 
alternate  network  paths.  ALIS  2.0.2  will  also  reduce  the 
need  for  propulsion  contractor  support  by  integrating  the 
first  portion  of  a  required  LLPM  capability. 

-  ALIS  3.0.0  will  complete  the  majority  of  the  ALIS 
development  effort.  The  schedule,  which  is  pending 
approval,  shows  a  fielding  date  of  July  2017.  This  version 
of  ALIS  will  include  a  complete  LLPM  capability  and 
eliminate  the  need  for  propulsion  contractor  support. 

•  The  following  sections  describe  progress  in  the  development 
and  fielding  of  ALIS  hardware  and  alignment  with  ALIS 
software  capabilities  described  earlier: 

-  The  program  continued  to  field  ALIS  hardware 
components  at  new  locations  during  CY14  as  the  global 
sustainment  bed-down  and  F-35  basing  continued  to  be 
activated.  The  table  on  the  following  page  shows  ALIS 
components,  location,  and  sustainment  function  for  new 
components  fielded  in  CY14. 

-  In  order  to  reduce  post-flight  data  download  times,  the 
program  added  and  fielded  a  new  piece  of  hardware,  the 
Portable  Maintenance  Device  (PMD)  reader,  to  operational 
units  beginning  in  July  2014.  The  PMD  reader  is  designed 
to  accelerate  the  download  of  unclassified  maintenance 
data  from  the  aircraft  without  the  need  for  a  secure 
facility.  The  PMD  reader  permits  maintenance  personnel 
to  download  maintenance  data  only,  vice  waiting  for  full 
portable  memory  device  download  from  the  aircraft  to  be 
processed  in  a  secure  facility  via  the  Ground  Data  Security 
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ALIS  HARDWARE  FIELDED  IN  FY14 


Component 

Location 

Function 

Central  Point  of  Entry 

EglinAFB 

One  per  country  to  provide  in-country 
and  software  and  data  distribution,  enable 
interoperability  with  government  systems 
at  national  level,  and  enable  ALIS  data 
connectivity  between  bases. 

Standard  Operating 
Unit  (SOU) 

Beaufort  AcademicTraining  Facility 
Italian  FACO 

Italian  PilotTraining  Center 

Australian  PilotTraining  Center 

Luke  AFB  PilotTraining  Center 

Nellis  AFB  57th  Fighter  Wing 
Netherlands  SOU  (at  Edwards  AFB) 

Supports  squadron-level  F-35  operations, 
including  maintenance,  supply  chain 
management,  flight  operations,  training,  and 
mission  planning. 

Base  Kit 

Nellis  AFB 

Edwards  AFB 

Centralizes  base  supply  for  bases  operating 
with  several  squadrons. 

LHD  Ship  Kit 

USS  Wasp 

Similar  to  a  squadron  kit  but  permanently 
installed  shipboard. 

Deployment  Kit 

Luke  AFB  PilotTraining  Center 

Short  of  a  full  squadron  kit  but  contains 
sufficient  hardware  to  support  four  aircraft. 

Will  become  a  squadron  kit  upon  delivery  of 
remaining  hardware. 

Depot  Kit 

Hill  AFB 

MCAS  Cherry  Point 

Similar  to  a  base  kit  but  geared  to  support 
depot  operations. 

Assembly  Receptacle  (GDR). 

Testing  of  the  PMD  could  not  be  done 
at  the  flight  test  center  because  the 
architecture  of  the  ALIS  supporting 
the  developmental  testing  aircraft  is 
not  production-representative.  The 
fielded  PMD  readers  have  functioned 
as  intended.  Maintenance  downloads 
generally  take  less  than  5  minutes 
using  a  PMD  reader,  while  the 
procedure  using  the  ground  data 
receptacle  -  which  downloads  all 
data  recorded  on  the  PMD  -  usually 
takes  an  hour,  delaying  access  to 
maintenance  information. 

-  SOU  Version  1  (SOU  VI),  the 
current  ALIS  unit-level  hardware 
configuration,  failed  to  meet  the 
deployability  requirement  in  the 
ORD  due  to  its  size,  bulk,  and 
weight.  The  program  is  developing 
a  deployable  version  of  the  SOU, 
deemed  SOU  Version  2  (SOU  V2). 

It  will  support  Block  2B,  Block  3i, 
and  Block  3F  aircraft,  and  is  needed  for  service  IOC 
dates.  It  will  be  incrementally  developed  and  fielded  with 
increasing  capability  over  the  next  several  years. 

■  The  first  increment  of  SOU  V2,  a  modularized  and 
man-portable  design  for  easier  deployability,  will  first  be 
made  available  to  Marine  Corps  for  IOC  in  2015.  This 
first  increment  aligns  SOU  V2  hardware  and  ALIS  2.0.1 
software  release.  The  program  plans  to  conduct  limited 
validation  and  verification  testing  of  the  ALIS  2.0. 1 
software  on  the  SOU  V2  once  delivered  to  Yuma  MCAS 
(planned  for  May  2015),  and  prior  to  fielding  it  to  other 
units  in  J  uly. 

■  The  second  increment  of  SOU  V2  went  on  contract  in 
August  2014.  This  increment  will  address  Air  Force 
hardware  requirements  for  sub-squadron  reporting 
capabilities  and  inter-squadron  unit  connectivity  and 
will  align  with  release  of  ALIS  software  version  2.0.2. 

It  is  scheduled  to  begin  testing  at  the  flight  test  centers  in 
July  2015. 

■  The  third  increment  of  SOU  V2,  which  also  went 
on  contract  in  August  2014,  will  address  hardware 
requirements  for  decentralized  maintenance,  which  will 
allow  maintenance  personnel  to  manage  tasks  with  or 
without  connectivity  to  the  main  SOU  and  allow  for  a 
Portable  Maintenance  Aid-only  detachment;  it  will  align 
with  ALIS  3.0.0. 

ALIS  was  designed  to  provide  the  analytical  tools  and 
algorithms  to  assess  air  vehicle  health  management  using 
health  reporting  codes  (HRCs)  collected  during  flight. 

These  functions  will  enable  the  Prognostic  Health  and 


Management  (PHM)  System  as  it  matures.  PHM  has 
three  major  components:  fault  and  failure  management 
(diagnostic  capability),  life  and  usage  management 
(prognostic  capability),  and  data  management,  all  of  which 
will  be  an  integral  part  of  ALIS.  Currently  PHM  has  no 
prognostic  capability,  while  diagnostic  and  data  management 
functionality  remain  immature.  The  program  plans  to 
include  the  first  set  of  prognostic  algorithms  in  ALIS  2.0.2. 

-  Diagnostic  capability  is  designed  to  enable  maintenance  by 
detecting  true  faults  within  the  air  vehicle  and  accurately 
isolating  those  faults  to  a  line-replaceable  component.  To 
date,  the  diagnostic  functional  capability  has  demonstrated 
low  detection  rates,  poor  accuracy,  and  high  false  alarm 
rates.  The  table  on  the  following  page  shows  metrics  of 
diagnostic  capability,  the  ORD  threshold  requirement  at 
maturity  (200,000  hours),  and  demonstrated  performance 
as  of  May  2014.  For  comparison,  demonstrated 
performance  from  May  2013  is  also  shown.  While 
detection  and  isolation  performance  metrics  improved 
between  May  2013  and  May  2014,  mean  flight  hours 
between  false  alarm  perfonnance  decreased  (worsened). 
-Asa  result,  fielded  operations  have  had  to  rely  on  manual 
workarounds,  such  as  maintainer-initiated  built-in  tests, 
extra  scheduled  inspections,  and  reliance  on  contractor 
support  personnel,  for  more  accurate  diagnostics  of 
system  faults.  Although  these  workarounds  have  aided 
troubleshooting,  they  increase  the  maintenance  man-hours 
per  flight  hour  and  reduce  sortie  generation  rates. 
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METRICS  OF  DIAGNOSTIC  CAPABILITY 


Diagnostic  Measure 

Threshold 

Requirement 

Demonstrated 
Performance 
(May  201 3) 

Demonstrated 
Performance 
(May  2014) 

Fault  Detection  Coverage  (percent  mission  critical  failures  detectable  by  PHM) 

98 

74 

81 

Fault  Detection  Rate  (percent  correct  detections  for  detectable  failures) 

98 

73 

81 

Fault  Isolation  Rate  (percentage):  Electronic  Fault  to  One  LRC 

90 

77 

72 

Fault  Isolation  Rate  (percentage):  Non-Electronic  Fault  to  One  LRC 

70 

70 

79 

Fault  Isolation  Rate  (percentage):  Non-Electronic  Fault  to  3  or  Fewer  LRC 

90 

80 

85 

Mean  Flight  Hours  Between  False  Alarm 

50 

0.59 

0.23 

Mean  Flight  Hours  Between  Flight  Safety  Critical  False  Alarm 

450 

69 

170 

Joint  Technical  Data  (JTD) 

•  Lack  of  verified  JTD  modules  continues  to  challenge 
fielded  operations,  requiring  workarounds  such  as  ARs 
to  the  contractor  for  engineering  dispositions  on  required 
maintenance  actions.  Also,  maintenance  personnel  in 
the  fielded  units  are  finding  that  verified  JTD  may  not  be 
adequate  to  complete  maintenance  actions  efficiently,  such  as 
an  engine  removal  and  replacement  and  maintenance  built-in 
test  troubleshooting. 

•  JTD  modules  are  first  identified  as  needed  in  the  field,  then 
developed  by  the  contractor,  and  finally  verified  before  being 
provided  to  the  operating  locations.  Entire  JTD  packages 
(i.e.,  all  JTD  modules  bundled  together)  are  periodically 
distributed  to  field  locations  using  ALIS,  and  then 
downloaded  at  the  units  to  the  Portable  Maintenance  Aids. 

-  The  current  process  is  cumbersome,  as  all  modules  are 
distributed  together,  including  modules  with  no  changes 
or  updates,  along  with  new  modules  and  those  with 
updates.  ALIS  2.0.0  should  allow  the  program  to  deliver 
partial  JTD  builds  (i.e.,  changes  and  amendments  to 
existing  JTD). 

-  The  total  number  of  data  modules  identified  continues  to 
grow  as  the  program  matures  and  additional  JTD  deliveries 
are  added  in  LRIP  contracts.  According  to  Program  Office 
schedules,  the  development  of  identified  JTD  modules  for 
each  variant  of  air  vehicle  and  for  propulsion  is  on  track 

to  meet  Service  milestones.  Air  vehicle  JTD  includes 
time  compliance  technical  data,  depot-level  technical  data, 
air  vehicle  diagnostics  and  troubleshooting  procedures, 
complete  structural  field  repair  series  data,  aircraft  battle 
damage  assessment  and  repair,  and  maintenance  training 
equipment.  Propulsion  JTD  development  is  nearly 
complete  and  on  schedule.  Development  of  Support 
Equipment  (SE)  JTD  lags  the  Program  Office  schedule 
by  9  percent  (approximately  200  modules  out  of  2,150 
identified),  primarily  due  to  the  lack  of  delivery  of  fault 
isolation  engineering  source  data. 

•  Verification  of  air  vehicle  JTD  modules  is  behind  and  has 
been  slowed  by  the  program’s  dependence  on  production 
aircraft  to  conduct  opportunistic  aircraft  verification  events. 
Because  priority  is  given  to  the  flight  schedule,  verification 


events  are  not  scheduled  and  require  support  from  the  field 
to  complete  JTD  verification.  The  program  has  identified 
more  air  vehicle  JTD  modules  during  the  last  year,  hence 
the  percentage  of  JTD  modules  verified  has  not  increased 
substantially  compared  to  what  was  reported  in  DOT&E’s 
FY13  Annual  Report.  To  reduce  the  number  of  unverified 
JTD  modules  at  Marine  Corps  IOC  declaration,  the  program 
should  provide  dedicated  time  on  fielded  aircraft  for  F-35B 
JTD  verification  teams. 

-  SE  JTD  verification  occurs  as  modules  are  developed 
and  released  in  ALIS,  so  it  lags  the  schedule  by  a  similar 
amount  as  module  development.  SE  assets  at  the 
training  units  at  Eglin  AFB  are  the  primary  source  for  SE 
verification. 

-  The  program  placed  Supportable  Low  Observable  (SLO) 
JTD  verification  on  contract  in  March  2014,  with  most 
verification  performed  using  desktop  analysis.  SLO  JTD 
verification  for  the  F-35B  is  nearly  complete.  Since  many 
of  the  SLO  modules  for  the  F-35A  and  F-35C  variants  are 
similar  to  those  for  the  F-35B,  the  program  expects  the 
verification  of  SLO  modules  for  those  variants  to  proceed 
on  schedule.  SLO  JTD  verification  data  were  not  available 
at  the  time  of  this  report;  progress  in  identification  and 
development  of  SLO  modules  is  reported  separately  in  the 
table  below. 

•  As  of  the  end  of  October,  the  program  had  verified 
approximately  84  percent  of  the  identified  air  vehicle  JTD 
modules  for  the  F-35A,  74  percent  for  the  F-35B,  and  62 
percent  for  the  F-35C.  The  table  on  the  following  page 
shows  the  number  of  JTD  modules  identified,  developed,  and 
verified  for  the  air  vehicle  (by  variant),  pilot  flight  equipment 
(PFE),  and  SE.  Overall,  67  percent  of  the  air  vehicle,  PFE, 
and  SE  identified  modules  have  been  verified.  Propulsion 
JTD  and  SLO  JTD  are  tracked  and  addressed  separately. 

•  Propulsion  JTD  are  current  as  of  April  2014.  More  current 
information  was  not  available  for  this  report.  Propulsion 
JTD  development  and  verification  has  proceeded  on  schedule 
and  the  Program  Office  considers  completion  by  the  end  of 
SDD  as  low  risk. 
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F-35  JOINTTECHNICAL  DATA  DEVELOPMENT  AND  VERIFICATION  STATUS 

Air  Vehicle,  Pilot  Flight  Equipment  (PFE),  and  Support  Equipment  (SE) 

(as  of  October  2014) 

Module 

Type 

Modules 

Identified 

Modules 

Developed 

Percent 

Identified 

Modules 

Developed 

Modules 

Verified 

Percent 

Identified 

Modules 

Verified 

F-35A1 

Unit-level 

4,658 

4,198 

90.1% 

3,893 

83.6% 

F-35B1 

Unit-level 

5,783 

5,221 

90.3% 

4,300 

74.4% 

F-35C1 

Unit-level 

4,799 

3,764 

78.4% 

2,949 

61 .5% 

Common2 
(all  variants) 

Unit-level 

322 

201 

62.4% 

142 

44.1% 

PFE 

Common 

337 

254 

75.4% 

250 

74.2% 

SE 

Common 

2,150 

945 

44.0% 

604 

28.1% 

TOTAL 

18,049 

14,583 

80.8% 

12,138 

67.3% 

1.  Includes  field- and  depot-level  JTD  for  Operations  and  Maintenance, 
on- and  off-equipment  JTD,  and  structured  field  repairs. 
2.  Includes  aircraft  JTD  for  general  repairs,  sealants,  bonding,  structured 
field  repairs,  and  non-destructive  investigation. 

F-35  JOINTTECHNICAL  DATA  DEVELOPMENT  AND  VERIFICATION  STATUS 

Propulsion 
(as  of  October  2014) 

Module 

Type 

Modules 

Identified 

Modules 

Developed 

Percent 

Identified 

Modules 

Developed 

Modules 

Verified 

Percent 

Identified 

Modules 

Verified 

Propulsion 

Engine  and 
Lift  Fan 

3,123 

2,988 

95.7% 

2,840 

90.9% 

F-35  JOINTTECHNICAL  DATA  DEVELOPMENT  AND  VERIFICATION  STATUS 
Supportable  Low  Observable  (SLO) 

(as  of  October  2014) 


Module 

Type 

Modules 

Identified 

Modules 

Developed 

Percent 

Identified 

Modules 

Developed 

Modules 

Verified 

Percent 

Identified 

Modules 

Verified 

SLO 

F-35A 

676 

180 

26.6% 

N/A 

N/A 

F-35B 

550 

52 

9.5% 

F-35C 

547 

52 

9.5% 

Common 

2 

0 

0.0% 

•  SLO  JTD  are  current  as  of  the  end  of 
October  2014.  SLO  JTD  are  tracked  under  a 
separate  contract  with  a  period  of  performance  of 
February  2014  through  April  2016.  The  Program 
Office  did  not  have  data  showing  verification  of 
SLO  JTD  modules  in  time  for  this  report. 

•  When  verified  JTD  are  not  available  or  not 
adequate  to  troubleshoot  or  resolve  a  problem 
with  an  aircraft,  maintenance  personnel  submit 
ARs  to  the  contractor.  These  ARs  are  categorized 
as  critical  (Category  1)  or  severe  (Category  2)  and 
sub-categorized  as  high,  medium,  or  low. 

-  The  contractor  prioritizes  and  responds  to  ARs 
through  an  engineering  team  (referred  to  as  the 
Lightning  Support  Team),  which  is  composed 
of  Service  and  contractor  personnel. 

-  As  of  October  15,  2014,  24  Category  1  ARs 
remained  open  while  617  Category  2  ARs  were 
open.  The  number  of  open  Category  1  ARs 
has  remained  relatively  flat  over  the  last  year, 
while  the  number  of  open  Category  2  ARs  has 
decreased  by  two-thirds  since  January  2014. 

Air-Ship  Integration  and  Ship  Suitability  Testing 

F-35B 

•  The  program  previously  completed  two  test 
periods  on  the  USS  Wasp  with  developmental 
test  aircraft,  one  in  October  2011  and  one  in 
August  2013.  These  periods  assessed  handling 
qualities  for  take-off  and  landing  operations  at 
sea,  and  were  used  to  develop  an  initial  flight 
operating  envelope  for  ship  operations.  ALIS 
was  not  deployed  to  the  ship,  and  very  limited 
maintenance  operations  were  conducted  (routine 
pre-  and  post-flight  inspections,  refueling 
operations,  etc.). 

•  The  Marine  Corps  began  making  plans  to  conduct 
another  test  period  on  the  USS  Wasp  in  May  2015 
to  assess  ship  integration  and  suitability  issues, 
using  non-instrumented  production  aircraft  and 
a  non-deployable  version  of  ALIS  (SOU  VI) 
installed  on  the  vessel.  This  deployment  was  originally  a 
part  of  the  Block  2B  OUE;  however,  it  is  being  re-scoped  to 
support  plans  for  the  Marine  Corps  IOC  later  in  2015. 

-  Up  to  six  production  aircraft  are  planned  to  be  used  for 
the  deployment.  These  aircraft  are  not  instrumented  (as 
test  aircraft  are)  and  will  allow  the  USS  Wasp  to  operate 
its  radars  and  communications  systems  in  a  representative 
manner  since  there  is  no  concern  with  electromagnetic 
interference  with  flight  test  instrumentation. 

-  The  flight  operations  will  not  be  representative  of  combat 
operations,  unless  the  flight  clearance  and  associated 
certifications  enabling  the  deployment  include  clearances 
for  weapons  carriage  and  employment.  These  clearances 
are  expected  at  fleet  release,  which  the  program  plans  to 
occur  in  July  2015,  after  the  deployment. 


-  Maintenance  will  be  mostly  military,  but  with  contractor 
logistics  support  in  line  with  expected  2015  shore-based 
operations,  such  as  contractors  for  propulsion  data 
downloads  after  each  flight.  Maintenance  will  be  limited 
to  that  required  for  basic  flight  operations,  staging 
necessary  support  equipment  for  engine  and  lift  fan 
removals  only  to  check  if  space  permits,  and  loading  and 
downloading  demonstrations  of  inert  ordnance  on  the 
flight  deck. 

-  These  limitations  are  not  representative  of  combat 
deployment  operations. 

•  The  Marine  Corps  and  Naval  Air  Systems  Command  began 
exploring  issues  that  would  arise  with  employing  more  than 
six  F-35B  aircraft  per  Air  Combat  Element  (ACE)  on  L-class 
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ships.  ACE  represents  the  mix  of  fixed-  and  rotary-wing 
aircraft  assigned  to  the  ship  to  conduct  flight  operations  in 
support  of  Marine  Corps  combat  objectives.  These  “heavy” 
ACEs  could  include  up  to  20  F-35Bs,  or  12  or  16  F-35Bs 
plus  MV-22Bs,  depending  on  the  specific  L-class  vessel. 
Through  these  exercises,  they  identified  issues,  many  which 
will  apply  to  standard-sized  ACE  operations  as  well.  These 
issues  include: 

-  The  currently-planned  service  maintenance  concept,  where 
few  components  will  be  repaired  underway  but  must  be 
sent  for  repair  back  to  a  depot  facility  or  to  the  Original 
Equipment  Manufacturer  (OEM)  may  not  be  achievable 
for  initial  fielding.  The  program  is  conducting  a  Level 

Of  Repair  Analysis  (LORA)  to  assess  the  feasibility  of 
repairing  components  at  the  Intermediate  level,  including 
onboard  CVN  and  L-class  ships. 

-  More  than  six  F-35Bs  in  the  ACE  will  require  a  more 
robust  engine  repair  and  resupply  process  than  for 
the  standard,  six  F-35B  ACE.  The  Services  are  still 
investigating  the  best  method  for  FI 35  engine  re-supply 
at  sea.  Work  continues  on  the  heavy  underway 
replenishment  station  and  a  re-designed  engine  storage 
container  that  can  survive  a  drop  of  18  inches  while 
protecting  the  engine  and  weighing  low  enough  to  be 
transferred  across  the  wire  between  the  supply  ship  and  the 
L-class  or  CVN  ship.  Adequate  storage  is  needed  for  the 
engines,  spare  parts,  and  lift  fans,  as  well  as  workspace  for 
engine  module  maintenance  within  the  small  engine  shops 
on  L-class  vessels. 

-  Moving  an  engine  container,  including  placing  an 
engine  in  or  taldng  one  out  of  the  container,  requires  a 
20,000  pound-class  forklift  and  cannot  be  concurrent  with 
flight  ops  since  this  item  is  required  to  be  on  the  flight  deck 
for  crash  and  salvage  purposes  while  flying.  Engines  can 
be  moved  around  on  a  transportation  trailer  once  removed 
from  the  container  to  enable  engine  maintenance  in  the 
hangar  bay  during  flight  operations. 

-  Adequate  Special  Access  Program  Facilities  (SAPF)  are 
required  for  flight  planning  and  debriefing  aboard  the  ship. 
Current  modification  plans  for  L-class  vessels  are  expected 
to  meet  the  requirements  for  a  six  F-35B  ACE,  but  would 
be  inadequate  for  an  operation  with  more  aircraft. 

-  Unlike  many  legacy  aircraft,  the  F-35B  needs  external  air 
conditioning  when  on  battery  power  or  an  external  power 
source.  Cold  fueling  operations,  when  the  engine  is  not 
turned  on,  will  thus  need  an  air  conditioning  cart.  For 
many  more  F-35B’s  in  the  ACE,  the  logistics  footprint 
will  have  to  increase  significantly  to  include  more  air 
conditioning  units  as  many  aircraft  are  refueled  cold  to 
achieve  efficient  operations. 

F-35C 

•  The  program  began  testing  the  redesigned  arresting  hook 
system  on  aircraft  CF-3  at  Patuxent  River  Naval  Air  Station 
in  February  2014.  This  test  aircraft  is  modified  with  unique 
instrumentation  to  monitor  loads  on  the  arresting  hook 


system  and  the  nose  landing  gear  for  catapult  launches 
and  arrested  landings.  The  structural  survey  testing  was  a 
pre-requisite  for  initial  carrier  sea  trials. 

-  Testing  encountered  significant  delays,  as  numerous 
deficiencies  were  discovered,  some  requiring  new 
engineering  designs.  Testing  was  planned  to  be  completed 
in  July,  to  support  deployment  to  a  CVN  for  the  first  set  of 
sea  trials.  The  following  problems  caused  delays: 

■  In  February,  a  hydraulic  leak  in  the  nose  landing  gear 
steering  motor,  caused  by  over-pressurization,  required  a 
redesigned  valve  and  halted  testing  for  10  weeks. 

■  Excessive  galling  of  the  arresting  hook  pitch  pivot  pin, 
which  required  a  redesigned  pin  made  of  titanium  and 
additional  inspections  after  each  landing. 

■  Damage  to  the  nose  landing  gear  shock  strut,  which 
required  down  time  for  repair 

-  The  structural  testing  was  partially  completed  in  two 
phases,  all  on  CF-3. 

■  Phase  one  completed  September  10,  2014,  and  consisted 
of  24  test  points  needed  to  clear  a  monitored  envelope 
for  carrier  landings.  Completion  of  phase  one  was 
necessary  for  CF-3  to  conduct  landings  on  a  CVN  in 
November. 

■  Phase  two  consists  of  20  additional  test  points  to 
clear  an  unmonitored  envelope  for  carrier  landings. 
Completion  of  phase  two  testing  would  allow  non-loads 
instrumented  test  aircraft  to  conduct  landings  on  a 
CVN.  Phase  two  work  was  ceased  on  September  25, 
with  1 7  of  20  phase  two  test  points  completed,  but  the 
program  waived  the  remaining  three  test  points  to  allow 
CF-5  to  participate  in  DT-1. 

•  Carrier-based  ship  suitability  testing  is  divided  into  three 
phases. 

-  The  first  phase,  DT-1,  consisted  of  initial  sea  trials  to 
examine  the  compatibility  of  F-35C  with  a  CVN  class  ship 
and  to  assess  initial  carrier  take-off  and  landing  envelopes 
with  steady  deck  conditions.  DT-1  was  conducted 
November  3  -  15,  2014;  it  was  initially  scheduled  to  begin 
in  July. 

■  Testers  accomplished  100  percent  of  the  threshold  test 
points  and  88  percent  of  the  objective  points  during 
deployment,  completing  33  test  flights  (39.2  flight  hours) 
and  124  arrested  landings,  of  124  attempts,  including 
one  night  flight  with  two  catapult  launches  and  two 
arrested  landings.  The  results  of  the  test  were  still  in 
analysis  at  the  time  of  this  report. 

■  No  other  aircraft  deployed  to  the  carrier,  except  transient 
aircraft  needed  for  logistical  support.  All  landings  were 
flown  without  the  aid  of  the  Joint  Precision  Approach 
Landing  System,  which  is  planned  for  integration 

on  the  F-35C  in  Block  3F.  No  ALIS  equipment  was 
installed  on  the  carrier.  The  test  team  created  a  network 
connection  from  the  ship  to  the  major  contractor  in 
Fort  Worth  to  process  necessary  maintenance  actions. 
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-  The  second  and  third  phases,  DT-2  and  DT-3,  consist 
of  ship-borne  operations  with  an  expanded  envelope 
(e.g.,  nighttime  approaches,  higher  sea  states  than 
observed  in  DT-1,  if  available,  and  asymmetrical  external 
stores  loading).  DT-2,  which  is  currently  planned  for 
August  2015,  will  expand  the  carrier  operating  envelope. 
The  third  set  of  sea  trials  is  planned  for  CY16. 

•  The  Navy  is  working  on  the  following  air-ship  integration 
issues,  primarily  for  carriers.  Each  of  the  following 
integration  issues  also  applies  to  F-35B  on  L-class  ships, 
with  the  exception  of  Jet  Blast  Deflectors  (JBDs): 

-  Due  to  the  higher  temperature  of  F-35  engine  exhaust 
compared  to  legacy  aircraft,  carrier  JBDs  need  at  least 
two  modifications.  A  cooling  water  orifice  modification 
enables  basic  operations,  but  additional  side  panel 
cooling  must  be  added  for  higher  afterburner  thrust 
catapult  launches.  The  Navy  is  accomplishing  these  full 
modifications  on  at  least  some  JBDs  on  USS  Abraham 
Lincoln  (CVN-72)  in  preparation  for  IOT&E  and  on  USS 
George  Bush  (CVN-77)  for  developmental  testing,  and 
performed  the  basic  orifice  modification  on  USS  Nimitz 
(CVN-68)  for  the  November  DT-1. 

-  The  Lockheed  Martin-developed  F-35  ejection  seat 
dolly  failed  Critical  Design  Review.  The  F-35  ejection 
seat  has  a  higher  center  of  gravity  than  legacy  seats  due 
to  supports  for  the  helmet-mounted  display,  and  in  the 
shipboard  environment  needs  to  be  securely  tied  down  in 
case  of  rolling  motion  from  high  sea  states.  The  Navy  is 
investigating  developing  less  expensive  adapters  to  the 
current  ejection  seat  dolly,  and  determining  what  seat  shop 
modifications  (if  any)  will  be  required  to  safely  tie  down 
the  dolly  when  a  seat  is  installed. 

-  Two  separate  methods  for  shipboard  aircraft  firefighting 
for  the  F-35  with  ordnance  in  the  weapon  bays  are  being 
developed,  one  for  doors  open  and  one  for  doors  closed. 
Each  will  consist  of  an  adapter  that  can  fit  to  the  nozzle  of 
a  standard  hose.  The  open  door  adapter  will  also  attach 

to  a  24-foot  aircraft  tow  bar  so  firefighters  can  slide  it 
underneath  the  aircraft  and  spray  cooling  water  up  into  the 
bay. 

■  Testing  of  a  prototype  open  bay  adapter  was  conducted 
in  October  and  included  use  on  an  AV-8B  hulk,  propane 
fires,  and  JP-8  pool  fires,  as  well  as  assessing  ordnance 
cooling  effectiveness.  Mobility  tests  of  the  rig  were  also 
performed  on  CVN  and  L-class  non-skid,  asphalt,  grass, 
dirt,  and  rough  terrain.  All  tests  indicate  that  the  adapter 
provides  sufficient  access  to  the  bay  for  water  spray,  and 
featured  sufficient  ease  of  use  to  place  the  adapter  where 
needed  quickly  in  all  environments. 

■  The  closed  door  adapter  will  consist  of  a  penetrating 
device  to  punch  through  the  fuselage’s  carbon  fiber  skin, 
secure  in  place,  and  hold  when  water  pressure  is  applied 
so  deck  personnel  can  then  back  away  from  the  fire. 

The  Navy  also  plans  to  test  closed  bay  door  firefighting 
testing  of  on-aircraft  lithium  ion  battery  fires. 


-  Work  on  noise  abatement  during  launch  and  recovery 
continues.  The  Navy  is  installing  sound  dampening 
material  in  the  highest  noise  level  areas  for  flight 
operations  on  the  USS  Abraham  Lincoln  (CVN-72)  during 
its  nuclear  refueling  and  overhaul,  and  the  Office  of  Naval 
Research  (ONR)  will  analyze  effectiveness  compared  to 
untreated  ships.  This  effort  will  not  involve  treatment  of 
all  work  areas,  however,  and  may  not  be  sufficient  to  allow 
conversational-level  speech  in  every  mission  planning 
space  during  flight  operations. 

-  The  need  for  improved  flight  deck  hearing  protection 
is  not  limited  to  the  F-35,  as  the  F-35  and  F/A-18E/F 
Super  Hornet  produce  similar  maximum  ground  noise  in 
afterburner  (149  decibels  for  the  F-35  and  150  decibels  for 
the  Super  Hornet). 

■  Based  on  an  assumed  F-35  noise  environment  of 

149  decibels  when  in  maximum  thrust  where  personnel 
are  normally  located,  53  decibels  of  attenuation  is 
required  to  enable  38  minutes  of  exposure  to  this 
worst-case  noise  per  day  before  long-term  hearing  loss 
ensues.  This  is  estimated  to  be  equivalent  to  60  launches 
and  60  recoveries. 

■  Current  expected  performance  for  triple  hearing 
protection  only  reaches  into  the  mid  40 ’s  decibels  of 
attenuation  though,  which  enables  less  than  10  minutes 
exposure  to  maximum  noise  before  the  daily  limit  is 
reached.  Workarounds  may  include  re-positioning 
launch  crew  personnel  and  tighter  administrative 
controls  for  exposure  times. 

-  The  unique  Integrated  Power  Package  (IPP),  and 
high-speed/low-thrust  engine  turn  capability  for 
maintenance  on  the  F-35,  may  introduce  new  concerns 
for  hangar  bay  maintenance.  The  Navy  plans  to 
investigate  the  impact  of  IPP  exhaust  emissions  on  hangar 
bay  atmosphere,  exhaust  temperature,  and  the  noise 
environment  produced,  to  determine  acceptable  hangar 
bay  maintenance  practices.  No  IPP  or  engine  turns  were 
conducted  during  the  DT-1  sea  trials. 

Progress  in  Plans  for  Modification  of  LRIP  Aircraft 

•  Modification  of  production  aircraft  is  a  major  endeavor  for  the 
program,  driven  by  the  large  degree  of  concurrency  between 
development  and  production,  and  is  a  burden  independent  of 
the  progress  made  in  developmental  testing.  Modifications 
are  dependent  on  the  production,  procurement,  and  installation 
of  modification  kits,  completed  either  at  the  aircraft  depot 
locations  or  at  the  field  units.  The  program  will  need  to 
provide  operationally  representative  Block  3F  operational  test 
aircraft  for  an  adequate  IOT&E. 

•  During  CY14,  the  Program  Office  and  Services  continued 
planning  for  modification  of  early  production  aircraft  to  attain 
planned  service  life  and  the  final  SDD  Block  3F  capability, 
including  the  production  aircraft  that  will  be  used  to  conduct 
operational  testing.  Planning  had  previously  focused  on 
modifying  aircraft  in  preparation  for  the  Block  2B  OUE 
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and  Marine  Corps  IOC,  planned  by  the  program  to  occur  in 
mid-2015.  This  created  challenges  in  obtaining  long-lead 
items  and  dock  availability  at  aircraft  depots,  and  maintaining 
adequate  aircraft  availability  to  maintain  pilot  currency 
while  eventually  modifying  all  operational  test  aircraft  into  a 
production-representative  Block  2B  configuration.  However, 
the  decision  to  not  conduct  the  Block  2B  OUE  allowed  the 
program  to  focus  on  Marine  Corps  IOC  aircraft  requirements, 
while  attempting  to  create  a  more  efficient  modification 
plan  for  operational  test  aircraft  to  achieve  the  Block  3F 
configuration. 

-  The  Program  Office  has  prioritized  Block  2B  associated 
modification  for  Marine  Corps  F-35B  IOC  aircraft  over 
operational  test  aircraft.  Because  manufacturers  could 
not  meet  the  schedule  demand  for  modification  ldts,  not 
all  of  the  operational  test  aircraft  will  be  in  the  Block  2B 
configuration  by  early  2015  when  the  planned  start  of 
spin-up  training  for  the  OUE  would  have  occurred,  as  was 
noted  in  the  FY13  DOT&E  Annual  Report. 

-  Modification  planning  has  also  included  early  plans  to 
ensure  operational  test  aircraft  scheduled  for  IOT&E  will 
be  representative  of  the  Block  3F  configuration.  However, 
these  plans  show  that  the  program  is  likely  to  face  the  same 
scheduling  and  parts  shortage  problems  encountered  in 
planning  for  Block  2B  modifications  of  the  operational  test 
aircraft. 

•  Upgrading  aircraft  to  a  Block  2B  capability  requires 
the  following  major  modifications:  mission  systems 
modifications;  structural  life  limited  parts  (SLLP),  referred 
to  as  Group  1  modifications;  F-35B  Mode  4  operations 
modifications,  which  include  a  modification  to  the  three 
Bearing  Swivel  Module  (3BSM)  to  allow  F-35B  aircraft 
to  conduct  unrestricted  Mode  4  operations;  OBIGGS;  and 
upgrades  to  AL1S  and  the  training  systems  to  fully  support 
Block  2B-capable  aircraft. 

-  The  program  maintains  a  modification  and  retrofit 
database  that  tracks  modifications  required  by  each 
aircraft,  production  break  in  of  modifications,  limitations 
to  the  aircraft  in  performance  envelope  and  service  life, 
requirements  for  additional  inspections  until  modifications 
are  completed,  and  operational  test  requirements  and 
concerns. 

-  The  program  uses  this  database  to  develop  and  update 
a  complex  flow  plan  of  aircraft  and  engines  through 
depot-level  modifications,  modifications  completed  by 
deployed  depot  field  teams,  and  those  completed  by 
unit-level  maintainers. 

-  The  current  depot  flow  plan  shows  that  none  of 
the  operational  test  aircraft  would  become  fully 

Block  2B-capable  by  January  2015,  and  only  7  of  14  will 
complete  the  necessary  modifications  by  July  2015,  which 
was  the  planned  start  date  of  the  Block  2B  OUE.  Block  2B 
modifications  would  finally  be  complete  on  all  operational 
test  aircraft  in  September  2016. 

•  Program  Office  modification  planning  for  Block  3F  IOT&E 
has  begun  and  shows  some  of  the  same  scheduling  pressures 


as  have  been  observed  for  Block  2B;  however,  these  would 
have  been  much  worse  if  the  OUE  were  conducted.  The 
depot  flow  plan  includes  a  seven-month  placeholder  to 
complete  all  modifications  to  bring  each  operational  test 
aircraft  to  a  Block  3F  configuration,  though  the  span  of 
time  required  to  complete  these  modifications,  including 
the  next  increment  of  structural  modifications  (SLLP  Group 
2),  is  unknown.  Additions  to  modification  packages  are 
possible  as  the  potential  for  discoveries  in  flight  test  still 
exists.  Although  the  program  has  prioritized  for  modification 
the  aircraft  planned  to  be  used  for  IOT&E,  the  Air  Force 
plans  for  at  least  12  F-35A  aircraft  to  be  available  for  IOC 
declaration  in  2016.  These  Air  Force  IOC  aircraft  will  be 
in  the  Block  3i  configuration  from  production  Lot  6  or  later, 
and  may  require  a  post-production  OBIGGS  modification, 
which  could  compete  for  resources  with  the  aircraft 
scheduled  for  IOT&E. 

•  Management  of  the  SLLP  Group  2  modifications  will  need 
to  be  handled  carefully  as  the  program  and  Services  prepare 
for  IOT&E.  If  the  program  does  not  schedule  SLLP  Group  2 
modifications  to  operational  test  aircraft  until  after  IOT&E  is 
completed,  495  flight  hours  must  remain  before  reaching  that 
life  limit  so  aircraft  can  fully  participate  in  IOT&E,  per  the 
approved  TEMP. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  program  and 
Services  have  been  addressing  the  redesign  and  testing  of  the 
OBIGGS  system,  but  performance  assessment  has  not  yet  been 
completed.  The  Program  Office  addressed  the  vulnerability  of 
the  electrical  power  system  to  ballistic  threats.  The  remaining 
recommendations  concerning  the  reinstatement  of  the  PAO 
shut-off  valve,  reinstatement  of  the  dry-bay  fire  extinguisher 
system,  design  and  reinstatement  of  fueldraulic  shut-off 
system,  improvement  of  the  Integrated  Caution  and  Warning 
system  to  provide  the  pilot  with  necessary  vulnerability 
information,  and  a  higher  resolution  estimate  of  time 
remaining  for  controlled  flight  after  a  ballistic  damage  event 
are  all  outstanding. 

•  FY 14  Recommendations.  The  program  should: 

1 .  Update  program  schedules  to  reflect  the  start  of  spin-up 
training  for  IOT&E  to  occur  no  earlier  than  the  operational 
test  readiness  review  planned  for  November  2017,  and  the 
associated  start  of  IOT&E  six  months  later,  in  May  2018. 

2.  The  program  should  complete  lab  testing  of  the  mission 
data  loads,  as  is  planned  in  the  mission  data  optimization 
operational  test  plan,  prior  to  accomplishing  the  necessary 
flight  testing  to  ensure  the  loads  released  to  the  fleet  are 
optimized  for  performance.  If  mission  data  loads  are 
released  to  operational  units  prior  to  the  completion  of  the 
lab  and  flight  testing  required  in  the  operational  test  plan, 
the  risk  to  operational  units  must  be  clearly  documented. 

3.  The  program  should  complete  the  remaining  three  Block  2B 
weapon  delivery  accuracy  (WDA)  flight  test  events  using 
the  currently  planned  scenarios  and  ensuring  full  mission 
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systems  functionality  is  enabled  in  an  operationally  realistic 
manner. 

4.  The  program  should  require  the  contractor  to  conduct 
rigorous  finite-element  analyses  to  assess  the  benefit  of  LSP 
application  for  the  F-35B  durability  test  article  and  for  the 
F-35B  FS496  bulkhead  redesign. 

5.  The  program  should  provide  adequate  resourcing  to  support 
the  extensive  validation  and  verification  requirements  for 
the  Block  3  VSim  in  time  for  IOT&E,  including  the  data 
needed  from  flight  test  or  other  test  venues. 


6.  To  accelerate  verification  of  JTD  modules,  the  program 
should  provide  dedicated  time  on  fielded  aircraft  for  F-35B 
JTD  verification  teams. 

7.  Extend  the  full-up  system-level  decontamination  test  to 
demonstrate  the  decontamination  system  effectiveness  in  a 
range  of  operationally  realistic  environments. 

8.  The  program  should  ensure  adequate  testing  of  ALIS 
software  upgrades  on  operationally-representative  hardware 
is  complete  prior  to  fielding  to  operational  units. 
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Global  Command  and  Control  System  -  Joint  (GCCS-J) 


Executive  Summary 

In  FY14,  Defense  Information  Systems  Agency  (DISA) 
development  of  Global  Command  and  Control  System  -  Joint 
(GCCS-J)  focused  on  implementing  high-priority  capability 
enhancements,  infrastructure  improvements,  and  software  defect 
corrections  to  both  the  GCCS-J  Global  (referred  to  as  Global) 
and  Joint  Operation  Planning  and  Execution  System  (JOPES). 

Global 

•  DISA  developed  Global  v4.3  to  move  the  baseline  towards 
a  more  flexible  and  service-oriented  architecture  and 
provide  high-priority  updates  to  the  Integrated  Command, 
Control,  Communications,  Computers,  and  Intelligence 
System  Framework;  Joint  Targeting  Toolbox  (JTT);  and 
Modernized  Integrated  Database  (MIDB). 

-  Operational  testing  of  Global  v4.3,  conducted  by  the 
Joint  Interoperability  Test  Command  (JITC)  in  June  and 
August  2013,  identified  several  critical  defects  relating 
to  the  MIDB  and  to  the  T-Sync  server,  both  of  which 
are  used  to  synchronize  targeting  data  between  different 
versions  of  the  MIDB  database. 

•  DISA  developed  Global  v4.3  Update  1  to  implement 
additional  high-priority  updates  and  improvements  to 
JTT  and  incorporate  fixes  to  critical  MIDB  defects.  JITC 
conducted  an  operational  test  on  Global  v4.3  Update  1  in 
August  2014. 

-  GCCS-J  v4.3  Update  1  is  effective  for  use  in  higher 
echelons  where  the  primary  interfaces  are  those  defined 
as  joint  critical. 

-  GCSS-J  v4.3  is  operationally  suitable. 

-  The  National  Security  Agency  (NSA)  discovered  nine 
major  cybersecurity  vulnerabilities  after  conducting  a 
cooperative  Blue  Team  assessment  of  the  GCCS-J  v4.3 
baseline  in  April  2014.  GCCS-J  v4.3  Update  1  is  not 
survivable  until  DISA  corrects  major  cybersecurity 
vulnerabilities  identified  by  the  NSA. 

•  DISA  approved  Global  v4.3  Update  1  fielding  in 
September  2014. 

JOPES 

•  DISA  developed  JOPES  v4.2.0.3  to  implement  the  required 
framework  for  interoperability  and  synchronization 
between  JOPES  and  Defense  Readiness  Reporting 
System  -  Strategic  (DRRS-S).  DRRS-S  will  replace 
Status  of  Resources  and  Training  (SORTS)  as  the  readiness 
reporting  system  of  record  following  successful  operational 
testing  of  DRRS-S. 

-  During  the  November  2013  operational  test  when  JOPES 
v4.2.0.3  was  interfaced  to  a  test  DRRS-S  system,  users 
identified  nine  critical  defects.  A  JOPES  regression  test 
with  DRRS-S  in  May  2014  supported  the  validation  and 
closure  of  all  nine  critical  defects. 
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-  During  operational  testing,  JOPES  v4.2.0.3  performed 
correctly  with  the  legacy  Global  Status  of  Resources  and 
Training  (GSORTS).  JOPES  v4.2.0.3  was  approved  for 
fielding  in  June  2014,  with  the  interface  to  remain  with 
GSORTS. 


System 

•  GCCS-J  consists  of  hardware,  software  (commercial 
off-the-shelf  and  government  off-the-shelf),  procedures, 
standards,  and  interfaces  that  provide  an  integrated  near 
real-time  picture  of  the  battlespace  necessary  to  conduct 
joint  and  multi-national  operations.  GCCS-J  consists  of 
a  client/server  architecture  using  open-systems  standards, 
government-developed  military  planning  software,  and  an 
increasing  use  of  World  Wide  Web  technology. 

•  GCCS-J  consists  of  two  components: 

-  Global  (Force  Protection,  Situational  Awareness, 
Intelligence  applications) 

-  JOPES  (Force  Employment,  Projection,  Planning,  and 
Deployment/Redeployment  applications). 

Mission 

Joint  Commanders  utilize  the  GCCS-J  to  accomplish  command 
and  control. 

Global 

•  Commanders  use  Global: 

-  To  link  the  National  Command  Authority  to  the 
Joint  Task  Force,  Component  Commanders,  and 
Service-unique  systems  at  lower  levels  of  command 

-  To  process,  correlate,  and  display  geographic  track 
information  integrated  with  available  intelligence  and 
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environmental  information  to  provide  the  user  a  fused 
battlespace  picture 

-  To  provide  Integrated  Imagery  and  Intelligence 
capabilities,  which  integrate  imagery  and  other  relevant 
intelligence  into  the  common  operational  picture  and 
allow  commanders  to  manage  and  produce  target  data 
using  the  JTT 

-  To  provide  a  missile  warning  and  tracking  capability 
•  Air  Operations  Centers  use  Global: 

-  To  build  the  air  picture  portion  of  the  common 
operational  picture  and  maintain  its  accuracy 

-  To  correlate  or  merge  raw  track  data  from  multiple 
sources 

-  To  associate  raw  Electronics  Intelligence  data  with  track 
data 

-  To  perform  targeting  operations 


JOPES 

•  Commanders  use  JOPES: 

-  To  translate  policy  decisions  into  operations  plans  to 
meet  U.S.  requirements  for  the  employment  of  military 
forces 

-  To  support  force  deployment,  redeployment,  retrograde, 
and  re-posturing 

-  To  conduct  contingency  and  crisis  action  planning 

Major  Contractors 

•  Govermnent  Integrator:  DISA 

•  Software  Developers: 

-  Northrop  Grumman  -  Arlington,  Virginia 

-  Leidos  -  Arlington,  Virginia 

-  Pragmatics  -  Arlington,  Virginia 


Activity 

Global 

•  JITC  and  the  Air  Force  conducted  Global  v4.3  testing  at 
multiple  echelons.  JITC  led  testing  at  the  higher  Combatant 
Command  echelon,  conducting  operational  testing  in 

June  and  August  2013,  to  support  the  DISA  Global  v4.3 
fielding  decision  in  September  2013. 

•  DISA  developed  Global  v4.3  Update  1  to  address 
deficiencies  identified  during  Global  v4.3  operational 
testing.  Global  v4.3  Update  1  provides  high-priority 
intelligence  mission  updates  to  the  Theater  Ballistic  Missile 
correlation  systems,  JTT,  and  MIDB.  The  update  also 
resolves  49  defects  affecting  other  parts  of  the  system  and 
implements  security  lockdown  scripts  and  Information 
Assurance  Vulnerability  Alert  updates. 

•  The  NSA  conducted  a  cooperative  Blue  Team  assessment  of 
GCCS-J  v4.3  baseline  in  April  2014. 

•  JITC  led  Combatant  Command-level  testing, 
conducting  operational  testing  of  Global  v4.3  Update  1 
from  August  13  -  21,  2014,  in  accordance  with  the 
DOT&E-approved  operational  test  plan. 

•  DISA  approved  Global  v4.3  Update  1  fielding  in 
September  2014. 

•  DOT&E  submitted  an  OT&E  report  on  GCCS-J 
Global  v4.3  Update  1  in  December  2014. 

JOPES 

•  JITC,  in  conjunction  with  DISA,  conducted  a  system 
acceptance  test/operational  test  of  the  DRRS-S  interface  to 
JOPES  from  November  12  -  22,  2013. 

•  JITC,  in  conjunction  with  DISA,  conducted  a  regression 
test  of  the  DRRS-S  v4.6.1  interface  to  JOPES  from 
May  12-23,2014. 

•  DISA  approved  JOPES  v4.2.0.3  fielding  in  June  2014. 

•  JITC  and  DISA  conducted  all  testing  in  accordance  with  the 
DOT&E-approved  Test  and  Evaluation  Master  Plan. 

•  DOT&E  did  not  issue  a  report  on  the  JOPES  v4.2.0.3 
system  acceptance  test/operational  test,  in  accordance  with 


DOT&E  Guidelines  for  Operational  Test  and  Evaluation  of 
Information  and  Business  Systems,  September  14,  2010. 

Assessment 

Global 

•  GCCS-J  v4.3  Update  1  is  effective  for  use  in  higher 
echelons  where  the  primary  interfaces  are  those  defined  as 
joint  critical. 

-  Further  operational  testing  is  required  to  determine  the 
effectiveness  for  lower  echelons,  such  as  Air  Operations 
Centers,  where  significantly  more  mission-critical 
interfaces  are  required.  Some  of  these  interfacing 
systems  will  need  to  be  updated  to  properly  interface 
with  the  MIDB  2.2,  using  interfacing  services  provided 
by  GCCS-J. 

-  The  46th  Test  Squadron  will  test  lower  echelon  interfaces 
in  September  through  November  2014.  Operational 
testing  for  lower  echelons  is  planned  for  FY17. 

•  The  Blue  Team  assessment  of  GCCS-J  v4.3  baseline  found 
nine  major  cybersecurity  vulnerabilities. 

-  Although  DISA  made  some  corrective  actions  to  the 
tested  baseline,  they  still  need  to  address  most  of 
the  vulnerabilities.  In  general,  these  vulnerabilities 
continue  in  a  Plan  of  Action  and  Milestones  status  for 
Global  v4.3  Update  1. 

-  GCCS-J  v4.3  Update  1  is  not  survivable  until  DISA 
corrects  major  cybersecurity  vulnerabilities  identified  by 
NSA. 

•  Independent,  threat-representative  adversarial  testing  for 
GCCS-J  v5.0  is  scheduled  in  January  2015. 

•  MIDB  version  2.2  corrects  critical  problems  that  existed 
in  the  MIDB  2.1  Patch  13  version  that  was  used  with 
GCCS-J  4.3. 

-  GCCS-  4.3  Update  1  was  effective  for  operational  use 
in  mixed  GCCS-J  v4.2.0.9/GCCS-J  v4.3  Update  1 
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environments  where  users  at  multiple  sites  need  to 
collaborate  on  targeting  tasks. 

-  Sharing  of  targeting  information  in  mixed  GCCS-J 
v4.2.0.9/GCCS-J  v4.3  Update  1  environments  occurs 
through  the  T-Sync  system,  which  is  needed  to  achieve 
MIDB  data  synchronization  across  the  different  MIDB 
versions.  Synchronization  between  different  versions  of 
the  MIDB  occasionally  displayed  slowness,  and  in  one 
case,  a  target  list  did  not  fully  replicate. 

-  Close  collaboration  between  operators  working  on 
different  versions  of  the  MIDB  to  build  a  combined 
target  list  will  be  needed  until  legacy  versions  of  GCCS-J 
Global  are  phased  out  in  the  FY17  timeframe. 

•  GCCS-J  v4.3  Update  1  is  operationally  suitable. 

-  Priority  3  defect  workaround  instructions  were 
accurately  updated  and  available  to  the  operational 
community. 

-  GCCS-J  v4.3  Update  1  servers  and  clients  demonstrated 
an  operational  availability  of  0.99. 

-  Users  rated  the  system  data  as  usable  and  useful, 
providing  recent,  relevant,  data. 

-  Users  felt  the  system  supported  task  accomplishment, 
agreed  it  was  better  than  the  legacy  system,  and  agreed  it 
was  sufficient  to  conduct  their  mission. 

-  The  help  desk  was  responsive  and  worked  all  resolvable 
GCCS-J  problems  to  closure.  Users  rated  online  help 
favorably. 

JOPES 

•  During  the  November  2013  operational  test  when  JOPES 

v4.2.0.3  was  interfaced  to  DRRS-S,  users  identified  nine 

critical  defects.  Users  found  the  JOPES/DRRS-S  interface 

to  be  unreliable  and  overly  complex  to  maintain.  Users 


also  noted  there  was  no  validated  process  to  detennine  if 
the  databases  were  out-of-sync,  and  the  process  to  re-sync 
the  databases  was  not  clear.  The  planning  community 
was  unable  to  use  the  DRRS-S  Planning  and  Execution 
Dashboards  to  support  plan  analysis.  Users  also  reported 
data  discrepancies  between  the  plan  data  in  JOPES  and  the 
data  in  DRRS-S.  Database  comparisons  confirmed  data 
accuracy  with  DRRS-S  failed  to  meet  the  Key  Performance 
Parameter  threshold. 

•  The  May  2014  JOPES  regression  test  with  DRRS-S 
supported  the  validation  and  closure  of  all  nine  critical 
defects.  The  JOPES-DRRS-S  interface  showed  significant 
improvement  in  its  ability  to  support  the  upcoming  DRRS-S 
operational  testing,  which  is  planned  for  May  2015. 

•  During  operational  testing,  JOPES  v4.2.0.3  performed 
correctly  with  the  legacy  GSORTS.  DISA  approved 
JOPES  v4.2.0.3  for  fielding  in  June  2014,  with  the  interface 
to  remain  with  GSORTS.  DRRS-S  will  replace  SORTS 

as  the  readiness  reporting  system  of  record  following 
successful  operational  testing  of  DRRS-S. 

Recommendations 

•  Status  of  Previous  Recommendations.  DISA  and  the  Defense 

Intelligence  Agency  successfully  addressed  all  previous 

recommendations. 

•  FY 14  Recommendations.  DISA  should: 

1.  Correct  the  nine  major  cybersecurity  vulnerabilities 
identified  by  the  NSA  during  the  cooperative  Blue  Team 
assessment  of  the  GCCS-J  v4.3  baseline. 

2.  Conduct  NSA  Red  Team  testing  at  a  Combatant  Command 
site  to  assess  detect,  react,  and  restore  capabilities  of 
GCCS-J  and  its  net  defenders. 
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Joint  Information  Environment  (JIE) 


Executive  Summary 

•  Following  the  establishment  of  requirements  by  a  Chairman 
of  the  Joint  Chiefs  of  Staff  White  Paper  and  Deputy  Secretary 
of  Defense  implementation  guidance  in  2013,  DOT&E  placed 
the  Joint  Information  Environment  (JIE)  on  test  and  evaluation 
oversight  in  August  2013. 

•  JIE  is  not  a  program  of  record,  and  to  date,  the  Defense 
Information  Systems  Agency  (DISA)  and  the  Services  have 
not  conducted  any  operational  testing  of  the  JIE  infrastructure 
or  components.  Furthermore,  the  operational  parameters 
required  for  DOT&E  to  review  and  evaluate  JIE  are  still  under 
development  by  U.S.  Cyber  Command. 

•  DOT&E  is  working  with  the  DISA  Test  and  Evaluation  Office 
to  plan  for  an  early  operational  assessment  of  JIE  in  FY15. 

Capability  and  Attributes 

•  The  JIE  is  envisioned  as  a  shared  information  technology 
construct  for  DOD  to  improve  physical  infrastructure,  increase 
the  use  of  enterprise  services,  and  centralize  the  management 
of  network  security.  The  Joint  Staff  specifies  the  following 
enabling  characteristics  for  the  JIE  capability: 

-  Transition  to  centralized  data  storage 

-  Rapid  delivery  of  integrated  enterprise  services  (such  as 
email) 

-  Real-time  cyber  awareness 

-  Scalability  and  flexibility  to  provide  new  services 

-  Use  of  common  standards  and  operational  techniques 

-  Transition  to  a  single  security  architecture 

•  The  DOD  plans  to  achieve  these  goals  via  the  following 
interrelated  initiatives: 

-  Implementation  of  Joint  Regional  Security  Stack 
(JRSS)  hardware  and  other  security  constructs  as  part 
of  a  single  security  architecture.  These  will  establish 
a  federated  network  structure  with  standardized  access 
and  authentication  management,  as  well  as  centralized 
defensive  cyber  operations  and  DOD  Information  Network 
defense. 

-  Consolidation  of  applications  and  data  into  centralized 
data  centers  at  the  regional  or  global  level,  which  are  not 
segregated  by  military  Service. 

-  Upgrade  of  the  physical  infrastructure  to  include 
Multi-Protocol  Label  Switching  (MPLS),  which 
enables  higher  bandwidth/throughput,  and  faster  routing 
capabilities. 

-  Establishment  of  enterprise  operation  centers  to  centralize 
network  management  and  defense. 

•  JIE  is  not  a  program  of  record,  but  is  being  managed  by  the 
DOD  Chief  Information  Officer  (CIO),  with  DISA  as  the 
principal  integrator  for  services  and  testing.  An  Executive 
Committee,  chaired  by  the  CIO,  U.S.  Cyber  Command, 
and  the  Joint  Staff  J 6,  provide  JIE  governance.  The  initial 
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implementation  of  the  JIE  has  begun  both  in  the  U.S. 
and  in  the  European  theater  with  the  establishment  of  the 
first  capabilities.  Installations  are  ongoing  in  Europe, 
but  implementation  and  cutover  dates  remain  uncertain. 
Additional  theaters  of  interest  are  the  Pacific,  Southwest  Asia, 
and  the  continental  United  States. 

Activity 

•  The  Chairman,  Joint  Chiefs  of  Staff  published  a  White 
Paper  on  the  JIE  in  January  2013  and  the  Deputy  Secretary 
of  Defense  published  implementation  guidance  for  JIE  in 
May  2013.  DOT&E  subsequently  put  the  JIE  initiative  on  test 
and  evaluation  oversight  in  August  2013. 

•  DISA  has  rescheduled  an  early  operational  assessment  of 
the  European  theater  capabilities  originally  planned  for 
March  2014  to  2QFY15  to  accommodate  the  engineering, 
installation,  and  implementation  of  the  initial  JRSS  and 
MPLS  capabilities.  DISA  reports  that  these  operational 
capabilities  will  be  only  partially  implemented  in  time  for 
the  first  operational  assessment;  DOT&E  plans  to  conduct 
an  additional  assessment  when  the  full  capabilities  are 
implemented.  The  availability  of  test  sites  for  JIE  and 
component  tests  are  limited  and  advanced  planning  for  future 
tests  is  not  fully  matured. 

•  In  FY14,  DISA  conducted  extensive  lab-based  testing  and 
installation/functional  testing  of  both  JRSS  and  MPLS  at 
the  DISA  facilities  at  Fort  Meade,  Maryland,  and  Joint 
Base  San  Antonio,  Texas.  While  installations  of  key  JIE 
infrastructure  continue  in  the  European  area,  training  and 
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development  of  operational  procedures  and  concepts  are 
ongoing.  The  JRSS  installation  at  Joint  Base  San  Antonio  is 
now  providing  some  services  to  support  both  Army  and  Air 
Force  network  operations,  but  has  not  been  fully  implemented 
as  yet.  No  operational  tests  have  been  conducted  of  JIE 
infrastructure,  components,  tactics,  procedures,  or  operational 
concepts  to  date,  but  DOT&E  continues  to  monitor  the 
development  of  key  test  plans  and  concepts. 

•  DISA  has  established  a  test  and  evaluation  working-level 
Integrated  Product  Team  in  which  DOT&E,  the  Services, 
USD(AT&L),  and  DOD  CIO  representatives  participate. 

Assessment 

•  No  operational  test  data  are  available  at  this  point. 

•  Developmental  and  laboratory  testing  continues  at  initial 
JRSS  sites  at  Joint  Base  San  Antonio,  Texas,  and  the  DISA 
Enterprise  Services  Lab  at  Fort  Meade,  Maryland.  To  date, 
testing  focuses  on  system  functionality  and  DISA  has  not  yet 
scheduled  full  cybersecurity  testing. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  DOD  CIO 
and  Director  of  DISA  are  addressing  the  previous 


recommendations  in  that  test  schedules  and  plans  continue  to 
be  prepared  for  anticipated  test  events  in  FY15  and  long-range 
and  overarching  test  strategies  are  being  developed. 

•  FY 14  Recommendations.  DISA  should: 

1 .  Examine  the  availability  of  cyber  range  resources  to 
augment  the  existing  physical  installations  available  for 
testing. 

2.  Continue  to  develop  an  overarching  test  strategy  that 
encompasses  not  only  the  upcoming  testing  of  JIE  in 
Europe,  but  also  defines  the  key  issues  and  concepts  to  be 
tested  in  subsequent  tests  and  assessments.  Such  a  plan 
should  address  the  following  areas  of  interest: 

■  Overarching  T&E  framework  and  critical  test  issues 

■  The  role  of  both  lab  and  fielded  equipment  tests  in 
resolving  those  critical  issues 

■  Estimated  schedules  for  test  events  and  key  issues  to  be 
tested 

■  Evaluation  criteria  and  any  relevant  implementation 
decisions  points 

■  Resources  required 

■  The  role  of  the  Services  and  Service-sponsored 
Operational  Test  Agencies 
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Joint  Warning  and  Reporting  Network  (JWARN) 


Executive  Summary 

.  <■  WiM  1)011 

•  The  Army  Test  and  Evaluation  Command  (ATEC) 
conducted  operational  testing  of  the  Joint  Warning 
and  Reporting  Network  (JWARN)  Web  Application 
during  the  Network  Integration  Evaluation 
(NIE)  14.1  from  October  3  to  November  17,  2013, 
at  Fort  Bliss,  Texas,  and  duiing  the  Ulchi  Freedom 
Guardian  14  exercise  from  August  17  -  29,  2014, 
in  the  Republic  of  Korea. 

•  The  JWARN  Web  Application  provides  an 
operationally  effective  tool  to  provide  nuclear, 
biological,  and  chemical  situational  awareness  and 
support  operational  decision  making  to  protect 
units  operating  10  or  more  kilometers  from  the 
initial  chemical,  biological,  radiological,  and/or 
nuclear  (CBRN)  release. 

•  The  JWARN  Web  Application  is  not  operationally 
suitable  due  to  the  lack  of  adequate  training  tools 
for  operators  to  maintain  the  high  level  of  operator 
proficiency  required  and  the  complexity  of  JWARN 
Web  Application  installation  and  configuration 
procedures. 

System 

•  The  JWARN  is  a  joint  automated  CBRN  warning,  reporting, 
and  analysis  software  tool  that  resides  on  joint  and  Sendee 
conunand  and  control  systems  including  the  Global  Command 
and  Control  System  (GCCS)  -  Army.  GCCS  -  Joint, 

GCCS  -  Maritime,  and  Command  and  Control  Personal 
Computer/Joint  Tactical  Common  Workstation. 

•  The  JWARN  Web  Application  is  a  web-based  application  that 
resides  on  the  Battle  Command  and  Control  System  server 
that  is  accessed  by  JWARN  operators  using  a  client  computer 
operating  within  the  Army  Capability  Set  13/14  network 
envir  onment.  It  has  been  modified  to  operate  in  the  Army 
Capability  Set  11/12  network  environment  The  JWARN  Web 
Application  can  also  be  installed  and  operate  on  a  stand-alone 
computer. 

•  JWARN  software  automates  the  NATO  CBRN  warning 
and  reporting  process  to  increase  the  speed  and  accuracy 
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JWARN  Web  Application  screen  shot  depicting  CBRN  hazard  prediction 
overlay  on  common  operational  map 


of  information  sharing  to  support  force  protection  decision 
making  and  situational  awareness. 

•  JWARN  uses  the  common  operating  picture  of  the  host 
conunand  and  control  system  or  computing  environment 
to  display  the  location  of  CBRN  events  and  the  predicted 
or  actual  location  of  hazards  to  support  the  commander’s 
situational  awareness  and  operational  decision  making. 

Mission 

JWARN  operators  support  the  commander’s  force  protection  and 
operational  decisions  by  providing  analysis  of  potential  or  actual 
CBRN  hazard  areas  based  on  operational  scenarios  or  sensor  and 
observer  reports;  identifying  affected  units  and  operating  areas: 
and  transmitting  warning  reports. 

Major  Contractor 

Northrop  Grumman  Mission  Systems  —  Orlando,  Florida 


Activity 

•  ATEC  conducted  operational  testing  of  the  JWARN 
Web  Application  within  the  Army  Capability  Set  13/14 
network  envir  onment  during  NIE  14.1  from  October  3  to 
November  17,  2013,  at  Fort  Bliss,  Texas.  Testing  was 
conducted  in  accordance  with  a  DOT&E-approved  test  plan. 


•  The  Army  Research  Laboratory  Survivability  Lethality 
Analysis  Directorate  conducted  a  cybersecurity  vulnerability 
assessment  of  the  JWARN  Web  Application  from 
June  24  —  27,  2014,  at  the  Tactical  Systems  Integration  Facility 
at  Aberdeen  Proving  Ground,  Maryland. 


JWARN 
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•  ATEC  conducted  operational  testing  of  the  JWARN 
Web  Application  within  the  Army  Capability  Set  11/12 
network  environment  and  a  stand-alone  version  of  JWARN 
during  the  Ulchi  Freedom  Guardian  14  exercise  from 
August  17  -  29,  2014  in  the  Republic  of  Korea.  ATEC 
conducted  the  test  during  an  operational  exercise,  which 
limited  the  ability  to  execute  the  test  in  accordance  with  the 
DOT&E-approved  test  plan.  This  did  not  preclude  collecting 
the  data  necessary  to  resolve  the  key  issues  that  the  test  was 
designed  to  address. 

Assessment 

•  The  JWARN  Web  Application  is  operationally  effective  to 
provide  CBRN  warning  reports  in  time  for  units  operating 
10  or  more  kilometers  from  the  initial  CBRN  release  to 
institute  force  protection  actions  before  encountering  CBRN 
hazards.  It  provides  enhanced  nuclear,  biological,  and 
chemical  situational  awareness  and  supports  operational 
decision  making  in  response  to  CBRN  threats  by  automating 
the  NATO  Allied  Technical  Publication-45  process  of  warning, 
reporting,  and  hazard  prediction.  During  operational  testing 

at  N1E  14.1,  the  JWARN  did  not  provide  an  audible  alert  on 
the  operator’s  computer  when  a  warning  report  was  received 
or  a  banner  alert  if  the  warning  was  received  from  another 
JWARN  operator.  The  alerting  capability  was  implemented 
and  successfully  demonstrated  during  the  Ulchi  Freedom 
Guardian  14  exercise  in  Korea. 

•  The  JWARN  Web  Application  is  interoperable  with  both 
Army  Capability  Set  11/12  and  13/14  network-operating 
environments. 

•  The  JWARN  Web  Application  is  not  operationally  suitable 
due  to  the  lack  of  adequate  training  tools  for  operators  to 


maintain  the  high  level  of  operator  proficiency  required  and 
the  complexity  of  JWARN  Web  Application  installation  and 
configuration  procedures.  Prior  to  testing  during  NIE  14.1, 
there  was  no  formal  system  administrator  training  on  installing 
JWARN  on  the  network  servers.  This  resulted  in  a  seven-hour 
delay  in  establishing  JWARN  functions  within  the  brigade  test 
unit. 

•  The  JWARN  Web  Application  software  is  reliable  and 
demonstrated  an  operational  availability  of  96  percent  during 
operational  testing  at  NIE  14.1. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  program 
manager  has  addressed  the  previous  recommendation  to 
field  computer-based  training  for  JWARN  on  Command  and 
Control  Personnel  Computer,  GCCS  -  Joint,  GCCS  -  Army, 
and  GCCS  -  Maritime.  However,  the  program  manager  has 
not  yet  implemented  computer-based  proficiency  training 

to  support  the  deployment  of  the  JWARN  Web  Application, 
which  requires  a  different  approach  to  keep  pace  with 
continuous  software  upgrades  and  the  various  web-based 
services  with  which  JWARN  interfaces.  The  computer-based 
training  includes  practical  exercises  and  tests. 

•  FY14  Recommendations.  The  Program  Office  should: 

1 .  Consider  utilizing  the  stand-alone  version  of  the  JWARN 
Web  Application  and  CBRN  operational  scenarios  of 
varying  complexity  for  use  as  a  training  tool  to  maintain 
operator  proficiency. 

2.  Develop  and  field  a  JWARN  Web  Application  installation 
wizard  to  minimize  errors  during  installation  of  the  JWARN 
Web  Application. 
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Key  Management  Infrastructure  (KMI) 


Executive  Summary 

•  The  Key  Management  Infrastructure  (KMI)  Program 
Management  Office  (PMO)  and  Joint  Interoperability  Test 
Command  (J1TC)  completed  the  KMI  Spiral  2  Test  and 
Evaluation  Master  Plan  (TEMP)  Addendum,  and  DOT&E 
approved  it  on  March  7,  2014. 

•  The  National  Security  Agency  (NSA)  Senior  Acquisition 
Executive  declared  a  KMI  program  deviation  on  August  29, 
2014,  due  to  missing  the  Acquisition  Program  Baseline’s 
Spiral  2,  Spin  1  software  release  date  in  July  2014.  The 
PMO’s  revised  release  date  will  be  January  31,  2015. 

•  JITC  conducted  an  operational  assessment  (OA)  of 
Spiral  2,  Spin  1  capabilities  and  the  new  KMI  tokens  in 
September  2014.  DOT&E  issued  a  classified  OA  report  in 
November  2014. 

•  The  OA  successfully  demonstrated  new  KMI  capabilities  for 
supporting  F-22  Raptor,  Advance  Extremely  High  Frequency 
and  Mobile  User  Objective  System  Satellite  systems.  Benign 
Keying,  Secure  Terminal  Equipment  enhanced  cryptographic 
cards,  new  tokens,  and  transition  procedures.  The  OA 
concluded  with  no  high-priority  discrepancies. 

•  While  the  OA  was  successful,  DOT&E  and  JITC  identified  the 
following  areas  for  improvement: 

-  The  KMI  system  executed  the  Secure  Software 
Provisioning  capability  as  designed;  however,  due  to 
inadequate  training  and  procedural  problems,  the  KMI 
staff  had  difficulty  uploading  and  titling  the  software 
packages  for  distribution  to  KMI  operating  accounts. 

Secure  Software  Provisioning  did  not  perform  properly  for 
file  uploads  and  downloads,  and  installation  procedures 
were  inadequate. 

-  The  NSA  and  Service  help-desk  manning  and  training 
observed  during  the  OA  is  inadequate  to  meet  KMI 
installation,  network,  and  daily  activities  for  Service 
worldwide  transition  and  rollout  of  the  Spiral  2,  Spin  1 
system. 

-  Additional  training  and  enhanced  standard  operating 
procedures  are  needed  at  the  KMI  sites  to  leverage 
the  automated  notifications  in  the  KMI  system.  Those 
procedures  need  to  be  refined  and  rehearsed  for  routine  and 
critical  situations. 

System 

•  KMI  is  intended  to  replace  the  legacy  Electronic  Key 
Management  System  to  provide  a  means  for  securely  ordering, 
generating,  producing,  distributing,  managing,  and  auditing 
cryptographic  products  (e.g.,  encryption  keys,  cryptographic 
applications,  and  account  management). 


KMI  Client  Host 
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AKP  -  Advanced  Key  Processor 

CIK  -  Crypto  Ignition  Key 

CLUAS  -  Card  Loader  User  Application  Software 

HAIPE  -  High  Assurance  Internet  Protocol  Encryptor 

PCMCIA  -  Personal  Computer  Memory  Card  International  Association 

Reinit  -  Reinitialization 


•  KMI  consists  of  core  nodes  that  provide  web  operations 
at  sites  operated  by  the  NSA,  as  well  as  individual  client 
nodes  distributed  globally  to  enable  secure  key  and  software 
provisioning  services  for  the  DOD,  intelligence  community, 
and  agencies. 

•  KMI  combines  substantial  custom  software  and  hardware 
development  with  commercial  off-the-shelf  computer 
components.  The  custom  hardware  includes  an  Advanced 
Key  Processor  for  autonomous  cryptographic  key  generation 
and  a  Type  1  user  token  for  role-based  user  authentication. 
The  commercial  off-the-shelf  components  include  a  client 
host  computer.  High  Assurance  Internet  Protocol  Encryptor 
(KG-250),  monitor,  keyboard,  mouse,  printer,  and  barcode 
scanner. 

Mission 

•  Combatant  Commands,  Services,  DOD  agencies,  other 
Federal  Government  agencies,  coalition  partners,  and 
allies  will  use  KMI  to  provide  secure  and  interoperable 
cryptographic  key  generation,  distribution,  and  management 
capabilities  to  support  mission-critical  systems,  the  DOD 
Information  Networks,  and  initiatives  such  as  Cryptographic 
Modernization. 

•  Service  members  will  use  KMI  cryptographic  products  and 
services  to  enable  security  services  (confidentiality,  non 
repudiation,  authentication,  and  source  authentication)  for 
diverse  systems  such  as  Identification  Friend  or  Foe,  GPS, 
Advanced  Extremely  High  Frequency  Satellite  System,  and 
Warfighter  Information  Network  -  Tactical. 


KMI  81 


F  Y  1  4  DOD  PROGRAMS 


Major  Contractors 

•  Leidos-  Columbia,  Maryland  (Spiral  2  Prime) 

•  General  Dynamics  Information  Assurance 
Division  -  Needham,  Massachusetts  (Spiral  1  Prime) 

•  BAE  Systems  -  Linthicum,  Maryland 


•  L3  Communications  -  Camden,  New  Jersey 

•  SafeNet  -  Belcamp,  Maryland 

•  Praxis  Engineering  -  Annapolis  Junction,  Maryland 


Activity 

•  The  PMO  and  JITC  completed  the  KMI  Spiral  2  TEMP 
Addendum,  and  DOT&E  approved  it  on  March  7,  2014.  The 
KMI  TEMP  Addendum  describes  the  test  and  evaluation 
strategy  to  support  planned  Spiral  2  program  activities.  The 
PMO  and  JITC  produced  the  TEMP  Addendum  to  align  the 
formal  test  program  with  the  PMO’s  implementation  of  an 
Agile  software  development  methodology.  The  PMO  is 
planning  four  software  releases  (one  spin  per  year)  that  will 
lead  to  a  Full  Deployment  Decision  by  April  2017. 

•  The  PMO  rolled  out  new  KMI  tokens  in  May  2014  to  reduce 
fault  modes  and  improve  reliability.  The  PMO  conducted 
reliability  growth  tests  to  evaluate  the  tokens,  and  the  JITC 
and  Service  representatives  evaluated  the  new  tokens  in 
Spiral  2,  Spin  1  Developmental  Test  and  Evaluation  (DT&E) 
events  in  June  and  July  2014. 

•  The  NSA  Senior  Acquisition  Executive  declared  a  KMI 
program  deviation  on  August  29,  2014,  due  to  missing  the 
Acquisition  Program  Baseline’s  Spiral  2,  Spin  1  software 
release  date  in  July  2014.  The  PMO’s  revised  release  date  is 
January  31,  2015. 

•  JITC  conducted  an  OA  of  Spiral  2,  Spin  1  capabilities  and  the 
new  KMI  tokens  in  September  2014. 

•  DOT&E  issued  a  classified  OA  report  in  November  2014. 

•  JITC  is  developing  plans  for  a  Spiral  2,  Spin  1  Limited  User 
Test  to  be  conducted  in  2QFY15  to  demonstrate  that  the  KMI 
system  operates  comparably  in  the  operational  environment 
as  it  did  in  the  OA’s  representative  environment,  and  to  gain 
Service  stakeholder  acceptance. 

Assessment 

•  Users  are  satisfied  with  the  existing  Spiral  1  performance  and 
capabilities,  and  the  overall  KMI  capability  is  significantly 
improved  and  stable. 

•  The  KMI  PMO  and  test  community  devised  a  sound  test 
approach  to  support  the  program’s  Agile  development 
methodology  and  planned  capability  releases,  resulting  in  the 
Spiral  2  KMI  TEMP  Addendum’s  approval. 

•  At  the  recommendation  of  DOT&E,  the  KMI  PMO  adopted 
and  implemented  automated  software  testing,  additional 
KMI  token  testing,  and  reliability  growth  efforts  that  yielded 
substantive  improvements  in  system  performance  and  stability 
as  observed  during  the  Spiral  2,  Spin  1  OA. 

•  In  the  government-led  DT&E-2  in  June  2014,  JITC  and 
Service  test  participants  identified  high-priority  deficiencies, 
and  the  KMI  PMO  directed  the  developer  to  correct  the 
problems  and  release  an  updated  baseline  to  the  KMI  system. 


•  The  subsequent  DT&E -2  retest  in  late  June  2014  identified 
additional,  high-priority  deficiencies  and  structural  problems 
in  the  KMI  database.  These  problems  were  exacerbated  by 
inadequate  schedule  allocation  before  and  after  the  DT&Es. 
The  PMO  resolved  problems  found  in  the  DT&E -2  retest  by 
mid-August  2014.  The  KMI  PMO  delayed  the  start  of  the  OA 
approximately  60  days  until  sufficient  regression  testing  was 
conducted  to  ensure  the  system  was  ready  to  move  to  the  next 
phase  of  testing. 

•  A  combination  of  22  operationally  representative  Air  F orce, 
Army,  Marine  Corps,  Navy,  and  civil  KMI  accounts 
participated  during  the  OA  at  geographically-dispersed  sites. 

•  The  OA  concluded  with  no  high-priority  discrepancies.  The 
OA  successfully  demonstrated  new  KMI  capabilities  for 
supporting  F-22  Raptor,  Advance  Extremely  High  Frequency 
and  Mobile  User  Objective  System  Satellite  systems,  Benign 
Keying,  Secure  Terminal  Equipment  enhanced  cryptographic 
cards,  new  tokens,  and  transition  procedures. 

•  While  the  OA  was  successful,  DOT&E  and  JITC  identified  the 
follows  areas  for  improvement: 

-  The  KMI  system  executed  the  Secure  Software 
Provisioning  capability  as  designed;  however,  due  to 
inadequate  training  and  procedural  problems,  the  KMI  staff 
had  difficulty  uploading  and  titling  the  software  packages 
for  distribution  to  KMI  operating  accounts.  Secure 
Software  Provisioning  did  not  perform  properly  for  file 
uploads  and  downloads,  and  installation  procedures  were 
inadequate. 

-  The  NSA  and  Service  help  desk  manning  and  training 
observed  during  the  OA  is  inadequate  to  meet  KMI 
installation,  network,  and  daily  activities  for  Service 
worldwide  transition  and  rollout  of  the  Spiral  2,  Spin  1 
system. 

-  Additional  training  and  enhanced  standard  operating 
procedures  are  needed  at  the  KMI  sites  to  leverage 
the  automated  notifications  in  the  KMI  system.  Those 
procedures  need  to  be  refined  and  rehearsed  for  routine  and 
critical  situations. 

•  JITC  assessed  interoperability  for  fill  devices,  end 
cryptographic  units,  and  the  Electronic  Key  Management 
System  infonnation  exchanges.  The  KMI  Spiral  2,  Spin  1 
system  is  on  pace  to  achieve  interoperability. 

•  Continuity  of  operations  planning  and  facility  preparations  are 
nearing  completion;  continued  efforts  are  necessary  to  refine 
and  test  those  capabilities  and  procedures. 
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•  JITC  did  not  evaluate  KMI  Spiral  2  cybersecurity  in  the  OA 
but  will  in  future  test  events  in  accordance  with  the  KMI 
TEMP  Addendum  approved  in  March  2014. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  KMI  PMO 
satisfactorily  addressed  the  four  FY13  recommendations. 

•  FY14  Recommendations.  The  KMI  PMO  should: 

1 .  Continue  to  improve  the  KMI  software  development  and 
regression  processes  rigor  to  identify  and  resolve  problems 
before  entering  operational  test  events. 

2.  Ensure  adequate  schedule  time  is  allocated  to  support  test 
preparation,  regression,  post-test  data  analysis,  verification 


of  corrections,  and  reporting  to  support  future  deployment 
and  fielding  decisions. 

3.  Develop,  codify,  and  distribute  standard  operating 
procedures  to  KMI  Storefront  operators  and  users  for 
functions  that  require  routine  to  critical  coordination  across 
the  enterprise. 

4.  Continue  to  verify  increased  KMI  token  reliability  through 
a  combination  of  laboratory  and  operational  testing  with 
automated  data  collection  from  system  logs  for  accurate 
reliability  and  usage  analysis. 

5.  Fully  execute  the  continuity  of  operations  plan  to  ensure 
procedures  and  redundant  facilities  are  adequate. 
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Public  Key  Infrastructure  (PKI) 


Executive  Summary 

•  The  FOT&Es  I  and  II,  conducted  in  January  2013,  revealed 
effectiveness  and  suitability  problems.  Although  no 
independent  operational  testing  has  been  completed  since 
then,  the  program  manager  has  been  actively  addressing  the 
requirements  definition  and  system  engineering  problems  that 
led  to  these  deficiencies,  while  making  program  personnel 
and  contract  management  process  changes  to  improve  the 
program.  An  expert  token  reliability  team  is  currently 
addressing  ongoing  token  reliability  problems  in  the  field. 

•  In  October  2013,  the  DOD  Public  Key  Infrastructure 
(PKI)  program  manager  notified  the  Milestone  Decision 
Authority,  then  DOD  Chief  Information  Officer  but  now 
USD(AT&L),  that  it  would  exceed  the  criteria  established  for 
a  critical  change  as  defined  in  Title  10,  United  States  Code 
(U.S.C.),  section  2445c,  and  would  be  unable  to  achieve  a 
Full  Deployment  Decision  (FDD)  within  five  years  of  the 
selection  of  the  preferred  alternative.  The  National  Security 
Agency  (NSA)  Senior  Acquisition  Executive  declared  a 
schedule-related  PKI  program  critical  change  in  October  2013, 
and  subsequently  a  cost-related  change  in  March  2014. 

•  In  July  2014,  USD(AT&L)  recertified  the  PKI  program  to 
Congress  in  accordance  with  10,  U.S.C.,  section  2445c(d), 
and  in  an  Acquisition  Decision  Memorandum  (ADM)  further 
approved  the  PKI  Program  Management  Office’s  (PMO’s) 
funding  obligation  authorities  for  $10  Million  through 
September  2014  to  ensure  no  disruption  of  PKI  Increment  2 
program  service  while  restructuring  following  the  critical 
change. 

•  Due  to  program  delays  resulting  from  the  critical  change,  the 
PKI  PMO  did  not  conduct  any  operational  testing  in  FY14. 

•  The  DOD  PKI  program  manager  has  drafted  an  Acquisition 
Strategy  that  focuses  the  remaining  Increment  2  Spirals 

(3  and  4)  on  15  user-prioritized  capabilities.  These  capabilities 
will  improve  Secret  Internet  Protocol  Router  Network 
(SIPRNET)  token  management  and  reporting,  improve 
system  availability,  and  will  provide  new  infrastructures  for 
the  provisioning  and  management  of  the  Non-secure  Internet 
Protocol  Router  Network  (NIPRNET)  Enterprise  Alternate 
Token  System  (NEATS)  and  certificates  to  Non-Person 
Entities  (NPEs)  (e.g.,  workstations,  web  servers,  and  mobile 
devices). 

System 

•  DOD  PKI  provides  for  the  generation,  production, 
distribution,  control,  revocation,  recovery,  and  tracking 
of  public  key  certificates  and  their  corresponding  private 
keys.  The  private  keys  are  encoded  on  a  token,  which  is  a 
credit-card  sized  smartcard  embedded  with  a  microchip. 


CA  -  Certificate  Authority  NlPRNet  -  Non-secure  Internet  Protocol  Router  Network 

CCEB  -  Combined  Communications  Electronic  Board  RA  -  Registration  Authority 

DMDC  -  Defense  Enrollment  Eligibility  Reporting  System  RAPiOS  -  Real-Time  Automated  Personnel  Identification  System 

DODIN  -  Department  of  Defense  Information  Networks  RCVS  -  Robust  Certificate  Validation  Service 

ECA  -  External  Certification  Authority  SIPRNet  -  SECRET  Internet  Protocol  Router  Network 

LRA  -  Local  Registration  Authority 

•  DOD  PKI  supports  the  secure  flow  of  information  across  the 
DOD  Information  Networks  as  well  as  secure  local  storage  of 
information. 

•  DOD  PKI  uses  commercial  off-the-shelf  hardware,  software, 
and  applications  developed  by  the  NSA. 

-  The  Defense  Enrollment  Eligibility  Reporting  System 
(DEERS)  and  Secure  DEERS  provide  the  personnel  data 
for  certificates  imprinted  on  NIPRNET  Common  Access 
Cards  and  SIPRNET  tokens,  respectively. 

-  DOD  PKI  Certification  Authorities  for  the  NIPRNET 
and  SIPRNET  tokens  reside  in  the  Defense  Information 
Systems  Agency  Enterprise  Service  Centers  in  Oklahoma 
City,  Oklahoma,  and  Mechanicsburg,  Pennsylvania. 

•  Increment  1  is  complete  and  deployed  on  the  NIPRNET. 

The  NSA  is  developing  PKI  Increment  2,  and  the  Defense 
Information  Systems  Agency  is  supporting  PKI  operations, 
enablement,  and  security  solutions. 

•  Increment  2  is  being  developed  and  deployed  in  four  spirals 
on  the  SIPRNET  and  NIPRNET.  Spirals  1  and  2  are  deployed, 
while  Spirals  3  and  4  will  deliver  the  infrastructure,  PKI 
services  and  products,  and  logistical  support  required  by  the 
15  user-prioritized  capabilities. 

Mission 

•  Military  operators,  communities  of  interest,  and  other 
authorized  users  will  use  DOD  PKI  to  securely  access, 
process,  store,  transport,  and  use  information,  applications, 
and  networks. 

•  Commanders  at  all  levels  will  use  DOD  PKI  to  provide 
authenticated  identity  management  via  personal  identification, 
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number-protected  Common  Access  Cards  or  SIPRNET  tokens 
to  enable  DOD  members,  coalition  partners,  and  others  to 
access  restricted  websites,  enroll  in  online  services,  and 
encrypt  and  digitally  sign  e-mail. 

•  Military  network  operators  will  use  NPE  certificates  to  create 
secure  network  domains,  which  will  facilitate  intrusion 
protection  and  detection. 


Major  Contractors 

•  General  Dynamics  C4  Systems  -  Scottsdale,  Arizona  (Prime) 

•  90Meter  -  Newport  Beach,  California 

•  SafeNet  -  Belcamp,  Maryland 

•  Red  Hat  -  Richmond,  Virginia 


Activity 

•  In  October  2013,  the  PKI  program  manager  notified  the 
Milestone  Decision  Authority  that  it  would  exceed  the 
criteria  established  for  a  critical  change  as  defined  in  Title 
10,  U.S.C.,  section  2445c,  and  would  be  unable  to  achieve 
an  FDD  within  five  years  of  the  selection  of  the  preferred 
alternative.  The  NSA  Senior  Acquisition  Executive  declared  a 
schedule-related  PKI  program  critical  change  in  October  2013, 
and  subsequently  for  cost  in  March  2014. 

•  The  program  was  unable  to  achieve  the  FDD  objective  date  of 
March  2013  established  in  the  Major  Automated  Information 
System  original  estimate  to  Congress.  The  one-year  breach 
occurred  in  March  2014,  and  the  five-year  breach  occurred  in 
April  2014. 

•  At  the  Defense  Acquisition  Executive  Summary  review  in 
April  2014,  the  PKI  program  manager  identified  problems 
with  PKI  requirements,  Acquisition  Strategy,  funding,  and 
schedule  supportability. 

•  In  May  2014,  the  PKI  PMO  conducted  initial  vendor 
developmental  tests  for  planned  Token  Management  System 
(TMS)  Release  3.0  enhancements. 

•  In  July  2014,  the  USD(AT&L)  recertified  the 
PKI  program  to  Congress  in  accordance  with 

Title  10,  U.S.C.,  section  2445c(d),  and  in  an  ADM,  which 
further  approved  the  PKI  PMO  funding  obligation  authorities 
for  $10  Million  through  September  2014  to  ensure  no 
disruption  of  PKI  Increment  2  program  service  while 
restructuring  following  the  critical  change. 

•  The  PKI  PMO  is  currently  revising  the  PKI  Acquisition 
Strategy  and  plans  to  complete  Spirals  3  and  4  by  3QFY17. 

•  The  PMO  is  also  updating  the  PKI  System  Engineering  Plan, 
Spiral  3  Test  and  Evaluation  Master  Plan  (TEMP)  Addendum, 
Life  Cycle  Sustainment  Plan,  and  Transition  Plan. 

•  In  late  September  2014,  the  USD(AT&L)  signed  a  PKI 
Increment  2  restructure  ADM  that  restored  the  PMO’s 
obligation  authorities  and  provided  directives  for  updating 
important-planning  documents,  including  the  Spiral  3  and  4 
TEMP  Addenda. 

•  The  PKI  PMO  did  not  conduct  any  operational  testing  in 
FY14.  JITC  will  examine  interoperability  and  information 
security  during  Limited  User  Tests  and  subsequent  FOT&E 
events  tentatively  scheduled  for  2015  and  later. 


Assessment 

•  The  PKI  PMO’s  contractor-led  TMS  Release  3.0 
developmental  test  and  evaluations  (DT&Es)  demonstrated 
increased  planning  and  execution  rigor.  The  PMO  is 
planning  additional  government  DT&Es  in  September  and 
November  2014,  but  will  not  conduct  TMS  operational  testing 
until  September  2015. 

•  The  FOT&Es  I  and  II,  conducted  in  January  2013,  revealed 
effectiveness  and  suitability  problems.  Although  no 
independent  operational  testing  has  been  completed  since  then, 
the  program  manager  is  addressing  the  requirements  definition 
and  system  engineering  problems  that  led  to  these  deficiencies, 
while  making  program  personnel  and  contract  management 
process  changes  to  improve  the  program’s  ability  to  achieve 
current  restructured  goals.  An  expert  token  reliability  team  is 
currently  addressing  ongoing  token  reliability  problems  in  the 
field. 

•  The  NSA  Information  Assurance  Directorate  (1AD)  continues 
efforts  to  improve  PKI  token  reliability.  The  PKI  PMO’s  token 
vendor  recently  developed  token  version  3.2  that  is  intended  to 
correct  several  known  faults  in  the  token’s  operating  system. 
However,  the  NSA  IAD  will  not  certify  the  3.2  token’s 
operating  system  prior  to  distributing  65,000  new  tokens  to  the 
Marine  Corps  and  Air  Force  by  the  end  of  September  2014. 

It  is  possible  more  may  be  distributed  before  the  next  token 
version  3.3  is  certified. 

•  System  reliability,  availability,  and  maintainability  of  the  core 
PKI  infrastructure  continue  to  present  problems  as  reported 
by  users  in  the  field.  The  PMO  has  implemented  changes  to 
improve  overall  system  reliability;  however,  these  changes 
have  not  been  independently  verified  through  operational 
testing. 

•  Currently,  the  draft  PKI  Spiral  3  TEMP  Addendum  is 
improved  but  still  has  missing  information,  including 
reliability  growth  curves  needed  for  planning  tests  to  assess 
improvements  in  the  reliability  of  SIPRNET  tokens  and 
supporting  PKI  infrastructure.  For  example,  token  inventory 
management,  reporting  tools,  and  processes  are  still  not  in 
place  and  associated  requirements  are  not  clearly  defined. 

With  infrastructure  in  DOD-wide  use  and  tokens  in  the 
hands  of  a  majority  of  SIPRNET  users,  and  with  the  need  for 
replacement  cards  for  a  large  fraction  of  users  whose  tokens 
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are  expiring,  there  is  clearly  a  need  for  a  robust  inventory 
logistics  management  system. 

•  The  DOD  PKI  program  manager  has  drafted  an  Acquisition 
Strategy  that  focuses  the  remaining  Increment  2  Spirals 

(3  and  4)  on  15  user-prioritized  capabilities.  These  capabilities 
are  intended  to  improve  S1PRNET  token  management  and 
reporting,  improve  system  availability,  and  will  provide  new 
infrastructures  for  the  provisioning  and  management  of  the 
NEATS  and  certificates  to  NPEs  (e.g.,  workstations,  web 
servers,  and  mobile  devices). 

•  NS  A I  AD  is  conducting  formal  token  certification  tests  for 
version  3.3  to  ensure  that  no  vulnerabilities  are  exposed. 

•  The  PKI  PMO  adopted  a  Spiral  3  and  4  approach  in  the 
program’s  Acquisition  Strategy  that  more  logically  aligns  with 
the  capability  development  and  testing  efforts.  Spiral  3  will 
include  the  TMS  3.0  through  6.0  releases,  and  Spiral  4  will 
include  separate  releases  for  NPE  and  NEATS. 

•  The  PKI  PMO,  Service  representatives,  and  test  community 
are  working  together  to  refine  the  schedule;  however, 
additional  effort  is  needed  to  establish  an  event-driven  test 
approach  (versus  a  schedule-driven  approach)  that  supports  the 
draft  Acquisition  Strategy. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  PKI  PMO 
satisfactorily  addressed  the  four  previous  recommendations. 


FY14  Recommendations.  The  PKI  PMO  should: 

1 .  Update  the  TEMP  in  accordance  with  the  redefined  PKI 
Increment  2  Acquisition  Strategy  to  prepare  stakeholders 
for  the  remaining  deliveries,  resource  commitments,  and 
test  and  evaluation  goals. 

■  Clearly  define  the  strategy  to  address  token  reliability 
and  growth  in  the  System  Engineering  Plan  and 

Spiral  3  and  4  TEMP  Addenda  to  ensure  SIPRNET  token 
fielding  decisions  are  informed  by  thorough  testing. 

■  Establish  a  reliability  growth  program  for  the  PKI 
system’s  infrastructure. 

■  Operationally  test  new  SIPRNET  token  releases  prior  to 
fielding  decisions. 

■  Develop  a  supportable,  resourced,  event-driven  schedule 
to  guide  both  the  capability  development  and  the  testing 
approach. 

2.  Create  a  transition  plan  defining  roles  and  responsibilities 
for  stakeholders  to  support  a  smooth  transition  and  ensure 
minimal  impact  to  PKI  operations  once  the  program  enters 
sustainment. 

3.  Define  and  validate  sustainment  requirements  for  PKI 
capabilities. 
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Network  Integration  Evaluation  (NIE) 


In  FY14,  the  Army  executed  two  Network  Integration 
Evaluations  (NIEs)  at  Fort  Bliss,  Texas,  and  White  Sands 
Missile  Range,  New  Mexico.  NIE  14.1  was  conducted  in 
October  and  November  2013  and  NIE  14.2  was  conducted  in 
April  and  May  2014.  The  purpose  of  the  NIEs  is  to  provide  a 
venue  for  operational  testing  of  Army  acquisition  programs, 
with  a  particular  focus  on  the  integrated  testing  of  tactical 
mission  command  networks.  The  Army  also  intends  the  NIEs 
to  serve  as  a  venue  for  evaluating  emerging  capabilities  that  are 
not  formal  acquisition  programs.  These  systems,  termed  by  the 
Army  as  “systems  under  evaluation”  (SUEs),  are  not  acquisition 
programs  of  record,  but  rather  systems  that  may  offer  value  for 
future  development. 

The  Army’s  intended  objective  of  the  NIE  to  test  and  evaluate 
network  components  in  a  combined  event  is  sound.  The  NIE 
events  should  allow  for  a  more  comprehensive  evaluation  of 
an  integrated  mission  command  network,  instead  of  piecemeal 
evaluations  of  individual  network  components. 

NIE  14.1 

During  NIE  14.1,  the  Army  executed  an  FOT&E  for  the  Joint 
Warning  and  Reporting  Network  (JWARN)  and  an  operational 
test  for  the  AN/PRC- 117G  radio.  The  Army  intended  to 
conduct  an  FOT&E  for  the  Command  Post  of  the  Future 
(CPOF);  however,  due  to  system  software  stability  problems 
discovered  during  the  pilot  test,  this  operational  testing  was  not 
executed.  The  Army  also  conducted  assessments  of  14  SUEs. 
Individual  articles  providing  assessments  of  JWARN  and  the 


AN/PRC-117G  radio  can  be  found  separately  in  this  Annual 
Report. 

NIE  14.2 

During  NIE  14.2,  the  Army  conducted  a  Multi-Service 
Operational  Test  and  Evaluation  for  the  Joint  Battle 
Command  -  Platform,  an  FOT&E  for  Manpack  radio,  an 
FOT&E  for  Shadow  Tactical  Unmanned  Aerial  System,  and  the 
first  phase  of  IOT&E  for  Nett  Warrior.  Individual  articles  on 
these  programs  are  provided  later  in  this  Annual  Report.  The 
Army  also  conducted  assessments  of  13  SUEs  during  NIE  14.2. 


NIE  ASSESSMENT 


NIE  14.1  and  14.2  were  the  sixth  and  seventh  such  events 
conducted  to  date.  The  Army  has  developed  a  systematic 
approach  to  preparing  for  and  conducting  NIEs,  applying  lessons 
learned  from  previous  events.  Overall,  NIEs  have  been  a 
satisfactory  venue  for  conducting  operational  tests  of  individual 
network  acquisition  programs. 

Operational  Scenarios  and  Test  Design.  The  Army  Test 
and  Evaluation  Command’s  Operational  Test  Command, 
in  conjunction  with  the  Brigade  Modernization  Command, 
continues  to  develop  realistic,  well-designed  operational 
scenarios  for  use  during  NIEs.  Additionally,  the  2d  Brigade, 

1st  Armored  Division,  as  a  dedicated  NIE  test  unit,  is  a  valuable 
resource  for  the  conduct  of  NIEs. 

The  challenge  for  future  NIEs  will  be  to  continue  to  develop 
new  and  more  taxing  operational  scenarios  to  reflect  future 
combat  operations.  Future  NIEs  should  include  more 
challenging  and  stressful  combined  arms  maneuvers  against 
regular  conventional  forces.  Such  scenarios  would  place 
greater  stress  on  the  tactical  network  and  elicit  a  more  complete 


assessment  of  that  network.  Within  resource  constraints, 
the  Army  should  continue  to  strive  to  create  a  demanding 
operational  environment  at  NIEs  similar  to  that  found  at  the 
Army’s  combat  training  centers. 

Balance  between  Testing  and  Experimentation.  There  are 
inherent  tensions  between  testing  and  experimentation  as  they 
each  have  somewhat  different  objectives  and  requirements  for 
exercise  control,  scenarios,  and  data  collection.  For  example, 
experimentation  tends  to  be  more  freewheeling  than  operational 
testing,  as  it  seeks  to  examine  possible  new  capabilities  and 
tactics  in  a  relatively  unconstrained  environment.  Operational 
testing,  on  the  other  hand,  requires  more  control  over  the  tactical 
environment,  as  testing  seeks  to  confirm  the  performance 
of  acquisition  systems  with  well-defined  requirements  and 
concepts  of  operation.  Furthermore,  experimental  items 
that  interact  with  systems  undergoing  operational  test  may 
negatively  affect  test  system  performance  and  confound  the  test 
results.  The  Army  must  continue  to  give  priority  to  operational 
test  objectives  at  NIEs  and  ensure  that  experimentation  and 
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training  demands  do  not  interfere  with  the  requirements  for 
adequate  operational  testing. 

The  Army  is  considering  devoting  one  NIE  a  year  to  operational 
testing  and  the  other  annual  NIE  to  experimentation  and  force 
development.  Such  an  approach  would  pay  dividends  by 
focusing  individual  event  design  on  the  specific  requirements  of 
testing  or  experimentation. 

Instrumentation  and  Data  Collection.  The  Army  should 
continue  to  improve  its  instrumentation  and  data  collection 
procedures  to  support  operational  testing.  For  example,  the 
Army’s  Operational  Test  and  Evaluation  Command  should 
devote  effort  towards  developing  instrumentation  to  collect 
network  data  for  dismounted  radios,  such  as  the  Manpack 
radio.  Additionally,  the  Army  needs  to  emphasize  the  use  of 
Real-Time  Casualty  Assessment  (RTCA)  instrumentation.  An 
essential  component  of  good  force-on-force  operational  testing, 
such  as  that  conducted  at  NIEs,  is  RTCA  instrumentation, 
which  adequately  simulates  direct  and  indirect  fire  effects 
for  both  friendly  and  threat  forces.  Finally,  the  Army 
should  continue  to  refine  its  methodology  for  the  conduct  of 
interviews,  focus  groups,  and  surveys  with  the  units  employing 
the  systems  under  test. 


Threat  Operations.  An  aggressive,  adaptive  threat  intent  on 
winning  the  battle  is  an  essential  component  of  good  operational 
testing.  The  Army  continues  to  improve  threat  operations  during 
NIEs,  particularly  with  respect  to  threat  information  operations 
such  as  electronic  warfare  and  computer  network  operations. 
NIEs  should  incorporate  a  large,  challenging  regular  force 
threat.  This  threat  should  include  a  sizeable  armored  force  and 
significant  indirect  fire  capabilities.  The  threat  force  should  also 
include  appropriate  unmanned  aerial  vehicles. 

Logistics.  The  Army  should  place  greater  emphasis  during  NIEs 
on  replicating  realistic  battlefield  maintenance  and  logistical 
support  operations  for  systems  under  test.  Field  Service 
Representative  (FSR)  support  plans,  maintenance  and  repair 
parts  stockage,  and  the  quantity  and  management  of  system 
spares  do  not  accurately  reflect  what  a  unit  will  observe  upon 
fielding.  Easy  access  to  and  over-reliance  on  FSR  support  results 
in  the  test  unit  not  having  to  realistically  execute  its  field-level 
maintenance  actions.  Failure  to  accurately  replicate  “real  world” 
maintenance  and  logistics  support  causes  operational  availability 
rates  and  ease  of  maintenance  to  be  overestimated  in  NIEs. 


NETWORK  PERFORMANCE  OBSERVATIONS 


The  following  are  observations  of  tactical  network  performance 
during  NIEs.  These  observations  focus  on  network  performance 
deficiencies  that  the  Army  should  consider  as  it  moves  forward 
with  integrated  network  development. 

Complexity  of  Use.  Network  components,  both  mission 
command  systems  and  elements  of  the  transport  layer,  are 
excessively  complex  to  use.  The  current  capability  of  an 
integrated  network  to  enhance  mission  command  is  diminished 
due  to  pervasive  task  complexity.  It  is  challenging  to  achieve 
and  maintain  user  proficiency. 

Common  Operating  Picture  (COP).  Joint  Publication  3-0, 
(Joint  Operations)  defines  a  COP  as  “a  single  identical  display 
of  relevant  information  shared  by  more  than  one  command 
that  facilitates  collaborative  planning  and  assists  all  echelons 
to  achieve  situational  awareness.”  With  current  mission 
command  systems,  units  have  multiple  individual  COPs  (e.g., 
for  maneuver,  intelligence,  and  logistics)  based  upon  the 
corresponding  mission  command  systems,  instead  of  a  single 
COP  that  is  accessible  on  one  system.  The  Army  is  seeking  to 
resolve  this  problem,  and  these  efforts  should  continue. 

Unit  Task  Reorganization  and  Communications  Security 
(COMSEC)  Changeover.  Operational  units  frequently 
change  task  organizations  to  tailor  for  tactical  missions.  The 
process  to  update  the  networks  to  accommodate  a  new  unit 
task  organization  remains  lengthy  and  cumbersome.  Similarly, 
COMSEC  changeover  is  a  lengthy,  burdensome  process,  which 
requires  each  individual  radio  to  be  manually  updated.  This 
process  typically  requires  in  excess  of  24  hours  for  a  Brigade 


Combat  Team  to  complete.  This  is  an  excessive  length  of  time 
for  a  unit  conducting  combat  operations. 

Armored  Brigade  Combat  Team  Integration.  The  challenge 
of  integrating  network  components  into  tracked  combat  vehicles 
remains  unresolved.  Due  to  vehicle  space  and  power  constraints, 
the  Army  has  yet  to  successfully  integrate  desired  network 
capabilities  into  Abrams  tanks  and  Bradley  infantry  fighting 
vehicles.  It  is  not  clear  how  the  desired  tactical  network  will  be 
incorporated  into  heavy  brigades. 

Infantry  Brigade  Combat  Team  (IBCT)  Integration. 

Integration  of  the  network  into  the  light  forces  will  also 
be  challenging  given  the  limited  number  of  vehicles  in 
the  IBCT.  Most  of  the  key  network  components,  such  as 
Joint  Battle  Command  -  Platform,  are  hosted  on  vehicles.  The 
challenge  of  linking  into  the  tactical  network  is  particularly  acute 
at  company  level  and  below,  where  light  infantry  units  operate 
dismounted.  Future  NIEs  should  examine  the  IBCT  tactical 
network,  which  has  not  been  addressed  to  date. 

Soldier  Radio  Waveform  (SRW)  Range.  Testing  at  NIEs 
continues  to  demonstrate  the  shorter  range  of  SRW  vis-a-vis 
the  legacy  Single  Channel  Ground  and  Airborne  Radio  System 
(SINCGARS)  waveform.  This  is  not  surprising  given  that  SRW 
operates  at  a  much  higher  frequency  than  does  SINCGARS. 
Higher  frequencies  have  shorter  ranges  and  are  more  affected 
by  terrain  obstructions.  NIE  test  units,  particularly  when 
operating  dismounted,  have  consistently  found  SRW  ranges  to  be 
unsatisfactory  in  supporting  tactical  operations  and  prefer  using 
SINCGARS  due  to  its  longer  range. 
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Dependence  on  FSRs.  Units  remain  overly  dependent 
upon  civilian  FSRs  to  establish  and  maintain  the  integrated 
network.  This  dependency  corresponds  directly  to  the  excessive 
complexity  of  use  of  network  components. 

Survivability.  An  integrated  tactical  network  introduces 
new  vulnerabilities  to  threat  countermeasures,  such  as  threat 


computer  network  attacks  and  the  ability  of  a  threat  to 
covertly  track  friendly  operations.  The  Army  should  continue 
to  improve  its  capabilities  to  secure  and  defend  its  tactical 
network.  In  particular,  the  Army  should  ensure  that  brigade-level 
cybersecurity  teams  are  appropriately  manned  and  trained. 
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AH-64E 


Executive  Summary 

•  The  Army  conducted  the  Lot  4  AH-64E  FOT&E  I  at 
Eglin  AFB,  Florida,  from  August  4  -15,  2014.  The  test 
included  training,  force-on-force  missions  in  an  operational 
Link  1 6  network,  adversarial  cybersecurity  testing,  and 
was  preceded  by  two  years  of  developmental  testing  that 
included  component  qualification,  joint  interoperability, 
cybersecurity,  and  live  fire  testing. 

•  FOT&E  I  was  adequate  and  demonstrated  that  Link  16 
enhances  the  operational  effectiveness  of  Lot  4  AH-64E 
units.  Lot  4  AH-64E  air  weapons  teams  found  small  target 
formations  more  quickly  using  Link  16  target  tracks  than 
when  using  other  onboard  sensors.  Air  weapons  teams, 
equipped  with  Link  16,  enhanced  overall  situational 
awareness  by  providing  battlefield  information  to  the  joint 
tactical  air  picture. 

•  Lot  4  AH-64E  enhancements  add  700  pounds  to  the  aircraft 
compared  to  the  Lot  1  AH-64E.  AH-64E  demonstrated  in 
testing  and  in  a  recent  unit  deployment  to  Afghanistan  that 
the  aircraft  can  meet  operational  performance  requirements 
at  6,000  feet  pressure  altitude  and  95  degrees  Fahrenheit  if 
permitted  to  use  all  available  engine  power. 

•  The  Lot  4  AH-64E  remains  operationally  suitable  and 
demonstrated  improvements  in  reliability,  availability,  and 
maintainability  compared  to  Lot  1  AH-64E  operational  test 
results. 

•  The  Lot  4  AH-64E  remains  as  survivable  as  the  Lot  1 
AH-64E  against  ballistic  threats.  Survivability  against 
infrared  threats  is  degraded  compared  to  the  Lot  1  AF1-64E. 
Radar  and  laser-warning  systems  degrade  pilot  situational 
awareness. 

•  Lot  4  AH-64E  aircraft  ballistic  vulnerability  and  personnel 
force  protection  is  comparable  to  the  Lot  1  AF1-64E. 

System 

•  The  Army  received  an  approved  mission  design  series 
change  renaming  the  AF1-64D  Apache  Block  III  to  AH-64E 
in  September  2012. 

•  The  AH-64E  is  a  modernized  version  of  the  AH-64D 
Attack  Helicopter.  The  Army  intends  to  sustain  the  Apache 
fleet  through  the  year  2040.  The  AH-64E  is  organized  in 
Attack/Reconnaissance  Battalions  assigned  to  the  Combat 
Aviation  Brigades.  Each  Battalion  has  24  aircraft. 

•  The  AH-64E’s  advanced  sensors,  improved  flight 
performance,  and  ability  to  integrate  off-board  sensor 
information  provide  increased  standoff  and  situational 
awareness  in  support  of  the  joint  force. 

•  The  major  Lot  1  AH-64E  capability  improvements 
included: 


-  The  AH-64E  aircrew’s  ability  to  control  the  flight  path 
and  the  payload  of  an  Unmanned  Aircraft  System 

-  Improved  aircraft  performance  with  70 ID  engines, 
composite  main  rotor  blades,  and  an  improved  rotor 
drive  system 

-  Enhanced  communication  capability,  which 
includes  satellite  communication  and  an  integrated 
communication  suite  to  meet  global  air  traffic 
management  requirements 

•  Lot  4  AH-64E  retains  Lot  1  capabilities  and  adds  hardware 
and  software  for  Link  16  network  participation. 

•  The  Army  acquisition  objective  is  to  procure 
690  AH-64E  aircraft:  634  remanufactured  and 
56  new-build  aircraft. 

Mission 

AH-64E-equipped  units  shape  the  area  of  operations  and 
provide  the  Joint  Force  Commander  and  Ground  Maneuver 
Commander  the  ability  to  defeat  the  enemy  at  a  specified  place 
and  time.  The  Attack/Reconnaissance  Battalions  assigned  to 
the  Combat  Aviation  Brigade  employ  the  AH-64E  to  conduct 
the  following  types  of  missions: 

•  Attack 

•  Movement  to  contact 

•  Reconnaissance 

•  Security 

Major  Contractors 

•  Aircraft:  The  Boeing  Company  Integrated  Defense 
Systems  -  Mesa,  Arizona 

•  Sensors  and  Unmanned  Aircraft  System  datalink:  Longbow 
Limited  -  Orlando,  Florida,  and  Baltimore,  Maryland 
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Activity 

•  The  Army  conducted  ballistic  testing  of  the  Lot  4  AH-64E 
Reduced-size  Crashworthy  External  Fuel  System  (RCEFS) 
in  May  2013  in  accordance  with  the  military  standard  for 
evaluation  of  fuel  tanks  on  rotary-wing  aircraft. 

•  The  46th  Test  Squadron  conducted  waveform  conformance 
testing  of  the  Small  Tactical  Terminal  Radio,  version  2.9.2  at 
Eglin  AFB,  Florida,  in  February  2014. 

•  The  Joint  Interoperability  Test  Command  completed  joint 
interoperability  testing  of  the  Small  Tactical  Terminal  Radio 
version  3 . 1 .2  at  Mesa,  Arizona,  in  May  2014.  The  AH-64E 
exchanged  the  required  Lot  4  Link  16  messages  to  joint 
participants. 

•  The  Army  conducted  a  cooperative  cybersecurity  assessment 
of  the  Lot  4  AH-64E  configuration  from  June  24  -  26,  2014,  at 
Redstone  Arsenal,  Alabama. 

•  The  Army  conducted  the  Lot  4  AH-64E  FOT&E  1  in 
accordance  with  the  DOT&E-approved  test  plan  from 
August  4  -  15,  2014,  at  Eglin  AFB,  Florida.  FOT&E  I 
consisted  of  a  unit  equipped  with  Lot  4  Apache  aircraft 
conducting  force-on-force  missions  against  a  dedicated 
opposing  force  and  supported  by  an  operational  Link  16 
network. 

-  AH-64E  Air  Weapons  Teams  equipped  with  two 
Lot  4  AH-64E  aircraft  flew  120  hours  conducting 

22  force-on-force  missions  under  varying  environmental 
conditions,  with  and  without  Link  16  targeting 
information,  against  small  (less  than  10  vehicles)  and  large 
(10  or  more  vehicles)  target  formations. 

-  AH-64E  Air  Weapons  Teams  provided  support  to  friendly 
maneuver  forces  in  vehicles,  dismounted,  and  aboard 

a  large  transport  watercraft.  Enemy  forces  employed 
fast-attack  craft,  a  large  mine-laying  boat,  armored 
vehicles,  mechanized  air  defense  vehicles,  dismounted 
infantry  with  small  arms  and  man-portable  air  defense 
systems,  mortar  teams,  and  technical  vehicles.  Both 
friendly  and  enemy  forces  were  instrumented  with 
Real-Time  Casualty  Assessment  equipment  to  ensure 
operational  realism. 

-  The  Army  Threat  Systems  Management  Officer 
conducted  an  adversarial  cybersecurity  assessment 
from  August  11  -  15,  2014.  The  test  team  investigated 
deficiencies  identified  during  the  IOT&E  in  2012,  and 
conducted  passive  scans  of  the  AH-64E  and  its  associated 
networks. 

•  The  Army  completed  system-level  ballistic  vulnerability 
and  personnel  protection  analyses  of  the  Lot  4  AH-64E  with 
RCEFS  and  the  latest  armor  configuration  and  provided  a  draft 
report  in  August  2014. 

•  The  Army  conducted  infrared  survivability  testing  in 
September  2014  at  Redstone  Arsenal,  Alabama.  The  test 
compared  the  susceptibility  of  the  Lot  4  AH-64E  equipped 
with  Aircraft  Survivability  Product  Improvements  (ASPI) 
with  the  susceptibility  of  the  Lot  4  AH-64E  without  ASPI  to 
infrared  threat  seekers. 


Assessment 

•  FOT&E  I  was  adequate  and  demonstrated  that  Link  16 
enhances  the  operational  effectiveness  of  Lot  4 
AH-64E-equipped  units. 

-  Air  Weapons  Teams  equipped  with  Link  16  enhanced  joint 
interoperability.  The  aircrews  exchanged  Link  16  message 
sets  indicating  their  location,  heading,  weapons,  and 

fuel  status  with  live  and  simulated  Air  Force  fighters  and 
command  and  control  aircraft. 

-  Lot  4  AH-64E  Air  Weapons  Teams  found  small-target 
formations  on  average  seven  minutes  faster  using  Link  16 
target  tracks  than  when  using  other  onboard  sensors.  Large 
target  formations  with  five  or  more  vehicles  were  detected 
just  as  quickly  with  other  onboard  sensors  as  when  using 
Link  16  data. 

-  Link  16  targeting  data  cluttered  the  aircrew’s  display  and 
increased  pilot  workload  when  five  or  more  targets  were 
present.  In  an  environment  with  less  than  five  targets, 

Link  16  targeting  data  aided  the  aircrew’s  target  acquisition 
and  reduced  pilot  workload.  Total  aircrew  workload 
during  the  test,  including  the  use  of  Link  16,  was  low. 

-  Lot  4  AH-64E  aircrews  used  the  Small  Tactical  Terminal 
Radio  to  participate  in  a  joint  Link  16  environment  with 
live  and  simulated  Air  Force  fighters  and  command  and 
control  aircraft.  The  Small  Tactical  Terminal  Radio 
experienced  no  critical  or  operational  mission  failures, 
remained  synchronized  with  the  network  87  percent  of  the 
time,  and  demonstrated  a  95  percent  message  completion 
rate. 

•  Air-to-Air-to-Ground  video  transfer  enhanced  the  Lot  4 
AH-64E  Air  Weapons  Team’s  situational  awareness.  The 
aircrews  transmitted  video  between  aircraft  in  flight  and  to  the 
maneuver  operations  center  on  the  ground.  Aircrews  provided 
favorable  feedback  on  the  video  quality  and  utility. 

•  The  Enhanced  Image  Intensified  Television  mode  of  the  Pilot 
Night  Vision  System  enhanced  performance  and  improved  the 
pilot’s  ability  to  see  light  sources  and  avoid  obstacles  at  night. 

•  The  adversarial  cybersecurity  assessment  found  that  a 
vulnerability  of  the  Apache  electronics  architecture  identified 
during  the  IOT&E  in  2012  has  been  addressed  and  identified 
new  cybersecurity  vulnerabilities  on  the  Lot  4  AH-64E  and 
interfacing  systems. 

•  The  Lot  4  AH-64E  remains  operationally  suitable  and 
demonstrated  improvements  in  reliability,  availability,  and 
maintainability  compared  to  Lot  1  AH-64E  operational 
test  results.  Transfer  of  in-flight  maintenance  data  to  a 
ground-based  maintenance  section  while  the  aircraft  is 

in  mission  profile  was  successful.  The  System-Level 
Embedded  Diagnostics  aided  in  aircraft  recovery  after  mission 
completion. 

•  The  Lot  4  AH-64E  remains  as  survivable  as  the  Lot  1  AH-64E 
against  ballistic  threats.  Survivability  against  infrared 
threats  is  degraded  compared  to  the  Lot  1  AH-64E.  Infrared 
threat  acquisition  ranges  are  unchanged  or  increased.  Flare 
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effectiveness  is  decreased  depending  on  the  threat  and  the 
flight  profile  of  the  aircraft. 

•  Consistent  with  the  IOT&E  evaluation,  radar-  and 
laser-warning  systems  were  not  effective  during  FOT&E  I 
and  degraded  pilot  situational  awareness.  Threat-warning 
systems  performed  poorly  and  are  not  effectively  integrated 
on  the  aircraft.  Aircrews  received  frequent  false  alarms,  had 
no  selective  volume  control  of  the  warning  systems,  and 
experienced  cluttered  or  conflicting  threat  displays.  Aircrews 
ignored  radar-  and  laser-warning  systems  that  continuously 
announced  inaccurate  threat  identifications. 

•  Lot  4  AH-64E  enhancements  add  700  pounds  to  the  aircraft 
compared  to  Lot  1  AH-64E. 

•  External  fuel  tanks  met  ballistic  survivability  requirements  and 
supported  all  FOT&E  I  missions.  The  RCEFS  revealed  no 
threat  of  sustained  fire  or  catastrophic  structural  failures. 

•  The  updated  system-level  vulnerability  and  force  protection 
assessments  for  the  Lot  4  AH-64E  showed  sustained  ballistic 
protection  of  the  aircraft  and  crew. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  has 
addressed  some  recommendations  from  the  FY12  Apache 
Block  III  Annual  Report.  The  following  recommendations 
have  not  been  fully  implemented: 

1 .  Consider  incorporating  improvements  to  current 

threat-warning  systems  as  they  are  developed.  Upgrade 


radar-  and  laser-warning  systems  and  provide  for  adjustable 
controls  for  each  warning  system. 

2.  Address  pilot’s  confidence  concerns  with  regard  to  the 
transmission  design.  Investigate  the  feasibility  of  alternate 
transmission  designs  that  provide  redundant  hydraulic  and 
electrical  power  in  the  event  of  loss  of  power  to  the  tail 
rotor. 

3.  Perform  a  structural  analysis  of  the  composite  main  rotor 
blades  to  better  understand  the  load-carrying  capabilities  of 
the  blade  that  was  damaged  during  ballistic  testing. 

FY14  Recommendations.  The  Army  should: 

1 .  Improve  infrared  countermeasures  performance,  upgrade 
radar-  and  laser-warning  systems,  and  improve  integration 
of  aircraft  survivability  equipment  on  the  Lot  4  AH-64E. 

2.  Address  demonstrated  cybersecurity  vulnerabilities.  Plan 
and  conduct  unconstrained  exploitation  of  vulnerabilities 
during  adversarial  cybersecurity  testing. 

3.  Modify  aircraft  performance  charts  and  aircraft  software  to 
allow  mission  planning  using  actual  engine  performance 
ratings. 

4.  Continue  development  of  Link  16  capabilities  and  conduct 
follow-on  testing  during  FOT&E  II. 

5.  Develop  procedures  to  establish  and  maintain  independent 
Link  1 6  training  networks. 
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AN/PRC-1 17G 


Executive  Summary 

•  The  Army  has  fielded  the  AN/PRC- 117G  radio  to  combat 
units  in  Afghanistan.  Testing  of  the  full  capabilities  in  a 
realistic  operational  environment  was  not  conducted  on 
the  AN/PRC- 1 1 7G  radio  prior  to  fielding.  DOT &E  placed 
the  AN/PRC- 117G  radio  on  oversight  on  October  4,  2012, 
and  directed  the  Army  to  conduct  an  operational  test  in 
calendar  year  2013. 

•  The  Army  Test  and  Evaluation  Command  conducted 
operational  testing  of  the  AN/PRC- 117G  as  part  of 
the  Network  Integration  Evaluation  (NIE)  14. 1  at 
Fort  Bliss,  Texas,  in  November  2013. 

•  During  the  operational  test,  a  Threat  Computer  Network 
Operations  Team  consisting  of  members  from  the  Army 
Research  Laboratory/Survivability  Lethality  Analysis 
Directorate  and  Threat  Systems  Management  Office  (TSMO) 
conducted  cybersecurity  assessments  on  the  AN/PRC-117G 
radio.  The  TSMO  conducted  an  electronic  warfare  campaign 
including  direction  finding  and  open-air  jamming  of  both  the 
Soldier  Radio  Waveform  (SRW)  and  Adaptive  Networking 
Wideband  Waveform  (ANW2). 

•  Asa  result  of  the  OT&E  conducted  in  November  2013, 
DOT&E  recommended  the  Army  evaluate  the  overall  network 
architecture  to  improve  the  range,  reliability,  and  survivability 
of  the  network  and  simplify  network  management. 

System 

•  The  AN/PRC-1 17G  radio  is  a  single  channel  voice  and  data 
radio  that  is  capable  of  operating  in  a  frequency  range  of 
30  Megahertz  to  2  Gigahertz.  Operational  configurations 
include  manpack,  vehicular-mounted,  or  base-station 
operations. 

•  The  primary  AN/PRC- 11 7G  waveform  is  the  ANW2,  which  is 
a  Harris  Corporation  proprietary  waveform. 

•  The  AN/PRC-1 17G  is  capable  of  simultaneously  transmitting 
both  Voice  over  Internet  Protocol  and  digital  data  on  a  single 


channel.  Digital  data  include  file  transfers,  chat,  streaming 
video,  and  position  location  reports. 

•  The  Army  procured  and  fielded  the  AN/PRC- 1 1 7G  as  a 
tactical  satellite  radio  and  to  provide  a  networking  radio 
bridge  capability  until  the  Manpack  Radio  and  Mid-Tier 
Networking  Vehicular  Radio  (MNVR)  programs  of  record  are 
available. 

Mission 

•  The  Army  intends  for  tactical  units  to  employ  the 
AN/PRC- 117G  as  a  data  radio.  Specifically,  the  ANW2 
allows  units  to  use  Internet  Protocol  routing  to  transmit 
medium  to  high  bandwidth  data  traffic  over  tactical 
Very-High  Frequency,  Ultra-High  Frequency,  and  L-band 
radio  networks. 

•  AN/PRC- 1 1 7G  will  be  an  interim  commercial  off-the-shelf 
solution  until  the  MNVR  is  developed  and  fielded.  The  Army 
intends  for  the  MNVR  to  replace  the  cancelled  Joint  Tactical 
Radio  System  Ground  Mobile  Radios  program. 

Major  Contractor 

Harris  Corporation  -  Rochester,  New  York 


Activity 

•  The  Army  is  purchasing  the  AN/PRC- 1 1 7G  as  a  commercial 
off-the-shelf  item  to  fill  a  capability  gap  for  a  tactical  digital 
radio.  With  the  October  2011  cancellation  of  the  Joint 
Tactical  Radio  System  Ground  Mobile  Radio  program,  the 
Army  sought  an  interim  solution  to  fill  Brigade  Combat 
Teams  as  a  part  of  Capability  Set  13.  The  Army  used  an 
existing  General  Services  Administration  contract  to  purchase 
the  AN/PRC- 117G. 

•  In  20 1 1 ,  the  Army  placed  a  $63  Million  order  for 
1 6,000  AN/PRC- 1 1 7G  radios. 


•  The  Army  has  fielded  the  AN/PRC-117G  radio  to  combat 
units  in  Afghanistan.  Testing  of  the  full  capabilities  in  a 
realistic  operational  environment  was  not  conducted  on 
the  AN/PRC- 11 7G  radio  prior  to  fielding.  DOT&E  placed 
the  AN/PRC- 11 7G  radio  on  oversight  on  October  4,  2012, 
and  directed  the  Army  to  conduct  an  operational  test  in 
calendar  year  2013. 

•  The  Army  Test  and  Evaluation  Command  conducted 
operational  testing  of  the  AN/PRC-1 17G  as  part  of  the 
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Army’s  NIE  14.1  at  Fort  Bliss,  Texas,  in  November  2013,  in 
accordance  with  a  DOT&E-approved  test  plan. 

•  As  part  of  the  operational  test,  a  Threat  Computer  Network 
Operations  Team,  consisting  of  members  from  the  Army 
Research  Laboratory/Survivability  Lethality  Analysis 
Directorate  and  TSMO,  conducted  cybersecurity  assessments 
on  the  AN/PRC- 11 7G  radio.  The  TSMO  conducted  an 
electronic  warfare  campaign  including  direction  finding  and 
open-air  jamming  of  both  the  SRW  and  ANW2.  All  threats 
portrayed  during  operational  testing  were  in  accordance 
with  the  accredited  Threat  Training  Support  Package  for  the 
AN/PRC- 117G  radio. 

•  DOT&E  published  an  Operational  Assessment  report  on  the 
AN/PRC- 117G  in  September  2014. 

Assessment 

During  the  NIE,  problems  with  the  network  architecture 
contributed  to  the  communications  problems  experienced  by 
the  test  unit.  The  AN/PRC- 1 17G-hosted  networks  were  able  to 
support  some  stationary  missions,  such  as  base  and  area  defense 
at  short  ranges  and  for  a  fraction  of  the  users.  Mobile  missions 
at  longer  ranges  presented  a  challenge  to  the  radio  networks.  A 
majority  of  Soldiers  reported  that  voice  communications  were 
acceptable. 

•  The  operational  ranges  for  AN/PRC- 11 7G  data  transfers 
were  too  short  to  support  their  combat  missions  at  echelons 
above  platoon;  as  designed,  the  network  cannot  support 
battalion-  and  company-level  communications  as  tactical  units 
require. 

•  No  requirements  document  exists  for  the  AN/PRC- 117G 
because  it  is  not  a  program  of  record.  The  operational 
test  conducted  during  the  NIE,  along  with  the  assessment 
conducted  in  Afghanistan,  demonstrated  the  radio  could 
not  meet  the  MNVR  Wideband  Networking  Waveform 
requirement  of  80  percent  “connection  availability”  at 
6-10  kilometers. 

•  The  operational  range  of  the  legacy  Single  Channel  Ground 
and  Airborne  Radio  System  (SINCGARS)  waveform 

on  the  AN/PRC- 11 7G  did  not  meet  Soldiers’  mission 
needs.  A  range  of  300  meters  was  reported  for  dismounted 
Soldiers,  and  2  kilometers  when  communicating  between  a 
dismounted  Soldier  and  a  vehicle-mounted  AN/PRC-1 17G. 
For  comparison,  a  legacy  SINCGARS  radio  demonstrated  a 
20-kilometer  range  during  the  Manpack  radio  Multi-Service 
Operational  Test  and  Evaluation. 

•  The  AN/PRC- 11 7G  demonstrated  a  long  Mean  Time  Between 
Essential  Function  Failure  (497  hours  for  the  SRW  and 
1,054  hours  for  the  ANW2),  which  indicates  a  low  failure  rate. 


•  During  the  NIE,  the  Soldiers  reported  the  following: 

-  The  size  and  weight  of  the  radio  made  it  portable. 

-  There  were  numerous  instances  of  the  radio  falling  out  of 
its  vehicle  mount. 

-  The  training  and  materials  provided  were  not  sufficient  for 
them  to  use  to  troubleshoot  and  repair  the  systems. 

•  Cybersecurity  vulnerabilities  on  the  AN/PRC- 117G 
permitted  access  to  networks  by  the  cyber  Red  Team.  These 
networks  contained  information  critical  to  Blue  Force 
operations.  During  the  operational  test,  the  Opposing  Force 
Commander  used  electronic  detection  of  Blue  Force  radios 
to  provide  situational  awareness  of  the  Blue  Force  locations. 
The  Opposing  Force  Commander  used  electronic  jamming 
to  disrupt  the  Blue  Force  scheme  of  maneuver.  Test  unit 
Soldiers  received  no  training  on  how  to  identify  and  respond  to 
electronic  warfare  or  cybersecurity  attacks. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed 
the  previous  recommendation. 

•  FY 14  Recommendations.  The  Army  should: 

1 .  Harden  the  radio  against  unauthorized  use  in  order  to 
prevent  the  cybersecurity  vulnerabilities. 

2.  Increase  the  transmission  range  of  ANW2  to  support 
operations  at  the  company  and  above  echelons  by  using 
a  lower  transmission  frequency,  increasing  antenna  size, 
and/or  increasing  output  power. 

3.  Improve  the  range  and  reliability  of  the  SINCGARS 
waveform.  The  performance  of  the  waveform  on  the 
AN/PRC- 117G  radio  should  be  comparable  with  the 
performance  of  the  legacy  SINCGARS  radio. 

4.  Provide  operations  and  maintenance  manuals  for  the 
AN/PRC- 117G  and  adequate  training  to  enable  unit 
Soldiers  to  operate  and  maintain  the  radio  under  normal 
operational  conditions  without  the  use  of  Field  Service 
Representatives.  Training  should  include  procedures  for 
identifying  and  responding  to  adversarial  electronic  and 
cybersecurity  attacks. 

5.  Improve  the  design  of  the  AN/PRC-117G  vehicle  mount 
to  prevent  the  radio  from  falling  out  of  the  mount  during 
vehicle  operations. 

6.  Evaluate  the  overall  network  architecture  to  improve  the 
range,  reliability,  and  survivability  of  the  network  and 
simplify  network  management. 
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C-17  Increased  Gross  Weight  (IGW)  and  Formation 

Spacing  Reduction  (FSR) 


Executive  Summary 

•  In  September  2013,  the  Army  Test  and  Evaluation  Command, 
with  support  from  the  Air  Force  Air  Mobility  Command, 
conducted  Increased  Gross  Weight  testing  for  a  single 

C-17  Globemaster  III  aircraft  with  a  full  paratrooper  load. 
This  was  Part  1  of  a  three-part  test. 

•  On  April  25,  2014,  DOT&E  published  an  Operational 
Assessment  report  on  Part  1 . 

•  Increasing  the  gross  weight  of  a  single  C-17  from 
385,000  to  400,000  pounds,  when  deploying  a  full 
paratrooper  load  with  the  T-ll  parachute  system  in  mass  exit 
configuration,  does  not  increase  the  risk  to  paratrooper  or 
aircraft  beyond  acceptable  levels  for  airborne  operations. 

•  Parts  2  and  3  of  the  test  effort  are  not  funded. 

System 

•  C-17  Increased  Gross  Weight  and  Formation  Spacing 
Reduction  are  proposed  changes  to  airborne  tactics, 
techniques,  and  procedures. 

•  The  Army  hypothesizes  that  the  introduction  of  the  new  T-ll 
parachute  may  allow  for  the  following  changes  in  order  to 
increase  unit  effectiveness  and  reduce  vulnerability  to  aircraft 
and  paratroopers: 

-  Increased  Gross  Weight:  Increase  C-17  gross  weight  by 
15,000  pounds  (from  385,000  to  400,000)  at  the  time  of 
a  parachute  drop.  This  will  allow  increased  range  for 
refueling  aircraft  and  projection  of  the  airborne  force 
further  into  a  hostile  environment. 

-  Formation  Spacing  Reduction:  Reduce  formation  spacing 
within  and  between  aircraft  elements  to  the  minimum  safe 
distance,  which  will  decrease  delivery  time  of  the  airborne 
force,  decrease  enemy  air  defense  reaction  time,  and 
increase  paratrooper  concentration  and  unit  cohesion  on 
the  ground. 

•  The  T-ll  Advanced  Tactical  Parachute  System  is  a  personnel 
parachute  system  consisting  of  the  main  parachute,  reserve 
parachute,  and  harness.  The  main  parachute  deployment 


system  uses  a  drogue/sleeve  and  a  mesh  slider  to  control  the 
rate  of  canopy  opening  and  minimize  aircraft  exit  interference 
problems.  The  T-ll  replaced  the  T-10  parachute. 

•  The  C-17  Globemaster  III  aircraft  is  one  of  the  Air  Force’s 
main  long-range,  heavy  transport  aircraft.  It  can  deploy 
102  paratroopers  from  two  troop  doors  in  a  single  pass. 

Mission 

Airborne  forces  execute  parachute  assaults  to  destroy  enemy 
forces  and  seize  and  hold  key  objectives  until  linkup  with 
follow-on  forces.  Airborne  assaults  are  used  in  forced  entry 
operations  to  deliver  Soldiers  with  speed  and  surprise  into  hostile 
territory. 

Major  Contractors 

•  T-ll: 

-  BAE  Systems  -  Phoenix,  Arizona 

-  Aerostar  International  -  Sioux  Falls,  South  Dakota 

-  Airborne  Systems  North  America  -  Santa  Ana,  California 

•  C-17:  The  Boeing  Company,  Integrated  Defense 
Systems  -  Long  Beach,  California 


Activity 

•  The  Army  Test  and  Evaluation  Command,  with  support  of 
the  Air  Force  Air  Mobility  Command,  executed  the  single 
C-17  Increased  Gross  Weight  test  in  September  2013  at 
Fort  Bragg,  North  Carolina.  This  was  the  first  of  a  three-part 
test. 

•  In  April  2014,  the  Army  and  Air  Force  began  collecting  and 
analyzing  vortex  data  to  populate  the  Vortex  Modeling  Tool 


that  will  be  used  in  the  development  of  C-17  formations  for 
Parts  2  and  3  of  the  test. 

•  DOT&E  published  an  Operational  Assessment  report  on  Part  1 
of  the  test  on  April  25,  2014. 

•  The  Army  and  Air  Force  conducted  all  testing  in  accordance 
with  DOT&E-approved  test  plans. 
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•  Parts  2  and  3,  which  will  test  multiple  aircraft  at  increased 
gross  weight  and  formation  spacing  reduction,  are  scheduled 
for  FY15. 

•  Required  Army  resources  for  Parts  2  and  3  of  the  test  effort  are 
not  funded. 

Assessment 

•  Increasing  the  gross  weight  of  a  single  C-17  from  385,000  to 
400,000  pounds,  when  deploying  a  full  paratrooper  load  with 
the  T-ll  parachute  system  in  mass  exit  configuration,  does  not 
increase  the  risk  to  paratroopers  or  aircraft  beyond  acceptable 
levels  for  airborne  operations.  During  the  test: 

-  Test  evaluators  did  not  observe,  nor  did  test  participants 
report,  any  Soldier-to-Soldier  interactions  after  exiting 
the  aircraft  until  parachute  stabilization  of  T-ll -equipped 
paratroopers  deploying  from  a  single  C-17  at  a  gross 
weight  of  400,000  pounds. 

-  Test  evaluators  did  not  observe  any  T-ll  parachute 
problems  at  400,000  pounds  aircraft  weight. 

-  There  was  no  damage  to  the  T-ll  parachute  system  caused 
by  deployment  from  a  C-17  at  400,000  pounds. 

-  Modeling  and  simulation  predicted  T-ll  -equipped 
paratroopers  deploying  from  C-17  at  400,000  pounds 
would  be  closer  during  parachute  deployment  than  T-ll 
or  T-10-equipped  paratroopers  at  385,000  pounds.  The 
jumper  separation  distance  predicted  is  better  than  the 
C-141  aircraft,  the  original  benchmark  for  paratrooper 
safety. 


•  The  current  airdrop  flight  profile  of  a  single  C-17  at 
385,000  pounds  can  be  maintained  at  400,000  pounds  while 
conducting  dual-door  mass  exit  operations  with  the  T-ll. 

•  Paratroopers  and  C-17  aircrews  used  current  single  aircraft 
tactics,  techniques,  procedures  and  training  to  execute  mass 
exit  operations  at  400,000  pounds. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations.  The  Army  and  Air  Force  should: 

1.  Populate  the  Vortex  Modeling  Tool  with  empirical  data  to 
support  C-17  aircraft  formation  spacing  at  increased  aircraft 
gross  weights  and  operational  jump  altitudes  (training  and 
combat)  with  different  environmental  conditions. 

2.  Progress  to  Parts  2  and  3  of  the  test,  multiple  aircraft  at 
increased  gross  weight  and  formation  spacing  reduction, 
after  completing  post-model  analysis  with  the  new  data. 

3.  Conduct  verification,  validation,  and  accreditation  of 
the  Vortex  Modeling  Tool  in  order  to  make  confident 
predictions  of  vortex  interactions  under  conditions  not 
included  in  the  test  effort,  such  as  combat  jump  altitudes. 

4.  Include  validation  testing  of  the  current  approved  formation 
geometry  at  an  aircraft  gross  weight  of  400,000  pounds 
during  Part  2  of  testing. 
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Distributed  Common  Ground  System  -  Army  (DCGS-A) 


Executive  Summary 

•  To  resolve  problems  discovered  during  the  IOT&E  in  2012, 
the  Army  reconfigured  the  system  as  Release  1  with  only 
the  Secret  enclave  components.  OSD  approved  the  full 
deployment  of  this  configuration.  The  Army  developed 
Release  2  to  address  the  capabilities  that  did  not  work 
effectively  in  Release  1,  to  include  re-adding  the  Top 
Secret/Sensitive  Compartmented  Information  (TS/SCI) 
enclave. 

•  The  U.S.  Army  Electronic  Proving  Ground  tested  Distributed 
Common  Ground  System  -  Army  (DCGS-A)  Release  2  in 
two  developmental  test  (DT)  phases: 

-  DTI :  A  lab  test  conducted  at  Aberdeen  Proving 
Ground,  Maryland,  from  January  30  through 
February  28,  2014.  The  Program  Office  conducted 
regression  testing  to  resolve  Priority  1  and  2  software 
problems,  which  were  discovered  during  DT  1 . 

-  DT2:  Conducted  September  13  -27,  2014,  at 
Fort  Huachuca,  Arizona,  using  a  representative 
operational  network. 

•  DOT&E  intends  to  issue  an  operational  assessment  of 
DCGS-A  based  on  the  DT2  results. 

•  DOT&E  will  make  a  full  evaluation  of  the  operational 
effectiveness,  operational  suitability,  and  survivability  of 
DCGS-A  Release  2  after  the  Army  completes  the  DCGS-A 
operational  test  planned  for  May  2015  during  the  Network 
Integration  Evaluation  15.2  event. 


ISR  Fusion  Server 
(in  Secret 
Collateral  enclave) 
with  NSA  tools 
(in  TS/SCI  enclave) 


Intelligence  DCGS-A  Geospatial 
Processing  Intelligence 

Center  Workstation 

(in  Secret  (in  Secret 

Collateral  enclave)  Collateral  enclave) 


Tactical  Ground 
Station 
(in  Secret 
Collateral  enclave) 


-  Workflows  that  are  based  on  how  an  intelligence  section 
would  employ  the  system 

-  Methods  for  transferring  entity  data  within  the  system  and 
between  systems  more  efficiently 

-  New  entity  database  structure 

-  Enhanced  fusion  software  for  correlation  of  entity  data 

-  New  geospatial  intelligence  configuration 

-  New  materiel  solution  for  transfer  of  information  across 
security  domains 

•  On  December  4,  2014,  USD(AT&L)  approved  modification 
to  the  acquisition  strategy  to  allow  Increment  1  to  end  with 
Release  2.  Requirements  that  were  allocated  to  Release  3,  to 
include  a  cloud  computing  capability  to  support  worldwide 
intelligence  analysis  and  database  synchronization,  including 
disconnected  or  low-bandwidth  environments,  will  now  be 
allocated  to  Increment  2. 


System 

•  DCGS-A  provides  an  organic  net-centric  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR)  capability  at 
the  brigade  level  by  combining  16  stove-piped  legacy 
applications  into  one  comprehensive  network,  including 
TS/SCI. 

•  To  resolve  problems  discovered  during  the  IOT&E  in  2012, 
the  Army  reconfigured  the  system  as  Release  1  with  only  the 
Secret-level  components.  OSD  approved  the  full  deployment 
of  this  configuration. 

•  The  Army  developed  Release  2  to  address  the  capabilities 
that  did  not  work  effectively  with  Release  1 .  Release  2  is 
intended  to  provide  enhanced  capabilities  to  include: 

-  TS/SCI  capability 


Mission 

Army  intelligence  analysts  use  DCGS-A  to  perform:  receipt  and 
processing  of  select  ISR  sensor  data,  intelligence  synchronization, 
ISR  planning,  reconnaissance  and  surveillance  integration,  fusion 
of  sensor  information,  and  direction  and  distribution  of  relevant 
threat,  non-aligned,  friendly  and  environmental  (weather  and 
geospatial)  information. 

Major  Contractors 

•  Lead  System  Integrator:  Intelligence  and  Information  Warfare 
Directorate,  U.S.  Army  Communications  -  Electronics 
Research,  Development,  and  Engineering  Center  -  Aberdeen 
Proving  Ground,  Maryland 

•  Northrop  Grumman  Electronic  Systems  -  Linthicum,  Maryland 


Activity 

•  From  January  30  through  February  28,  2014,  the  Army 
conducted  DTI  for  DCGS-A  Increment  1,  Release  2  at 
Aberdeen  Proving  Ground,  Maryland. 


The  Army  planned  DCGS-A  DT2  in  response  to 
the  December  14,  2012,  Full  Deployment  Decision 
Acquisition  Decision  Memorandum  that  required  “a  plan 
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for  a  developmental  test  with  a  representative  operational 
test  network  structure  using  the  scenarios  and  data 
collection/reduction  tools  expected  to  be  used  for  the 
operational  test.” 

•  The  Army  conducted  DT2  in  Fort  Huachuca,  Arizona,  from 
September  13  -  27,  2014.  The  Program  Office  resolved  all 
Priority  1  and  2  software  problems  before  starting  DT2. 

•  The  Electronic  Proving  Ground  will  publish  a  DT2  test  report, 
and  the  Army  Evaluation  Center  (AEC)  will  publish  a  DT2 
assessment  report.  Further  analysis  of  the  DT2  data  may  lead 
to  the  requirement  for  additional  corrective  actions. 

Assessment 

•  After  DTI,  the  Program  Office  continued  regression  testing  for 
software  fixes  delivered  through  multiple  fix  cycles. 

•  There  are  cybersecurity  issues  that  need  to  be  addressed. 
Cybersecurity  testing  was  conducted  during  the  last  week  of 
DT2  to  provide  an  update  to  the  cybersecurity  status. 

•  There  are  121  Priority  3  and  293  Priority  4  software  problems. 
All  of  these  will  need  a  workaround  written  and  trained.  The 
program  is  working  on  these  actions  but  if  not  completed  in 

a  timely  fashion,  the  performance  in  operational  test  could  be 
affected. 


•  As  of  December  4,  2014,  the  AEC  has  yet  not  used  DT2  data 
to  assess  key  DCGS-A  measures  of  performance  in  order 

to  help  evaluate  system  performance.  As  of  this  report,  it  is 
not  clear  the  quality  and  quantity  of  data  from  AEC  will  be 
sufficient  to  evaluate  key  measures  allocated  to  DT2  in  the 
approved  Test  and  Evaluation  Master  Plan.  Any  unanswered 
measures  must  be  addressed  during  the  Limited  User  Test 
planned  for  2015. 

•  DOT&E  will  publish  an  operational  assessment  report 
based  on  the  DT2  results,  and  will  make  a  full  evaluation 
of  the  operational  effectiveness,  operational  suitability,  and 
survivability  of  DCGS-A  Release  2  after  the  Army  completes 
the  DCGS-A  Limited  User  Test  planned  for  May  2015  during 
the  Network  Integration  Evaluation  15.2  event. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed 
all  previous  recommendations. 

•  FY 14  Recommendation. 

1 .  The  Army  should  conduct  the  Limited  User  Test 
incorporating  the  lessons  learned  from  the  DT2. 


102 


DCGS-A 


F  Y  1  4  ARMY  PROGRAMS 


DOD  Automated  Biometric  Identification  System  (ABIS) 


Executive  Summary 

•  The  Program  Manager  Biometrics  fielded  the  DOD  Automated 
Biometric  Identification  System  (ABIS)  1.0  to  the  Biometrics 
Identity  Management  Activity  (BIMA)  in  January  2009  as  a 
quick  reaction  capability  to  support  storing,  matching,  and 
sharing  of  collected  biometric  data  primarily  obtained  during 
Operation  Iraqi  Freedom  and  Operation  Enduring  Freedom. 

•  The  Army  chartered  the  Program  Management  Office  (PMO) 
in  2007  to  foster  the  establishment  of  ABIS  as  a  formal 
program  of  record  to  be  known  as  the  Biometrics  Enabling 
Capability  (BEC)  Increment  0. 

•  In  January  2011,  USD(AT&L)  issued  an  Acquisition  Decision 
Memorandum  establishing  ABIS  1 .2  as  the  baseline  for  the 
BEC  0  upon  completion  of  a  Full  Deployment  Decision 
(originally  scheduled  for  FY11). 

•  In  October  2014,  the  PMO  deployed  ABIS  1.2  successfully 
and  it  remains  the  authoritative  source  for  biometric 
transactions  upon  completion  of  the  two-phased  IOT&E  that 
was  conducted  from  August  through  October  2014. 

•  Prior  to  the  IOT&E,  the  Army  Test  and  Evaluation 
Command  (ATEC)  performed  a  customer  test  in 
February  and  March  2014.  The  test  operated  at  multiple 
sites,  with  the  primary  site  being  the  BIMA  facility 
located  in  Clarksburg,  West  Virginia.  The  purpose 

of  this  test  was  to  independently  verify  the  system 
readiness  of  the  DOD  ABIS  1 .2  system  prior  to  an 
operational  test.  Upon  completion  of  the  customer  tests, 

ATEC  conducted  a  two-phased  operational  test,  with 
Phase  One  held  August  7  -  28,  2014,  and  Phase  Two  held 
October  17-22,  2014. 

•  An  independent  Red  Team  assessment  in  August  2014 
revealed  significant  cybersecurity  vulnerabilities  that  must  be 
addressed.  ATEC  has  planned  a  cybersecurity  assessment  to 
be  held  during  the  2015  FOT&E  to  demonstrate  resolution  of 
critical  cybersecurity  findings. 

System 

•  The  DOD  ABIS  is  an  authoritative  database  that  uses  software 
applications  to: 

-  Process  and  store  biometrics  modalities  (i.e.,  fingerprints, 
palm  prints,  iris  scans,  and  facial  recognition  data)  from 
collection  assets  across  the  globe 

-  Update  the  biometric  database  repository  with  new 
biometrics  data 

-  Produce  biometrics  match  results  (against  stored  data) 

-  Share  responses  among  approved  DOD,  interagency,  and 
multi-national  partners,  in  accordance  with  applicable  law 
and  policy 

-  Provide  tools  to  monitor  the  health  and  status  of  the  system 


fighter  (JIM)  Vessel 


•  For  biometric  submissions  that  are  unable  to  produce  a  match 
using  automated  processes,  biometric  examiners  (subject 
matter  experts)  use  ABIS  workstations  with  specialized 
software  to  attempt  to  manually  match  submissions. 

•  ABIS  interfaces  with  global  biometrics  data  collectors  and 
users,  as  well  as  outside  databases. 

-  Military  Services  and  Combatant  Commands  collect 
biometrics  data  (fingerprint,  palm  print,  iris  scans,  and 
facial  scans)  from  persons  of  interest  in  the  field  using 
portable  collection  devices  and  submit  these  data  to  ABIS. 

-  Intelligence  analysts  analyze  and  fuse  biometrics 
information  via  the  Biometric  Identity  Intelligence 
Resources,  an  automated  database  outside  the  ABIS,  and 
provide  information  back  to  the  users  in  the  field. 

•  ABIS  1.2  uses  a  set  of  commercial  off-the-shelf  and  custom 
components  including: 

-  A  transaction  manager  for  managing  customer  submission 
workflows 

-  A  portal  allowing  authorized  operators  to  perform  user 
management,  system  configuration,  real-time  system 
monitoring,  submission  tracking,  and  report  generation 

•  The  U.S.  Army  BIMA  currently  operates  ABIS  on  the  DOD 
Non-secure  Internet  Protocol  Router  Network  (NIPRNET). 

•  The  PMO  developed  ABIS  1 .2  as  an  enhancement  to  the 
previously  fielded  version,  ABIS  1.0.  The  new  system  is 
intended  to  address  hardware  and  software  obsolescence  and 
scalability  limitations  in  ABIS  1.0,  and  increased  throughput 
and  storage  capacity  of  biometric  submissions  and  responses. 


DOD  ABIS 
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Mission 

•  Military  Services  and  U.S.  Combatant  Commands  rely  on 
AB1S  to  provide  timely,  accurate,  and  complete  responses 
indicating  whether  persons  of  interest  encountered  in  the  field 
have  a  prior  history  of  derogatory  (e.g.  criminal)  activity, 

to  assist  in  identifying  potential  threats  to  U.S.  forces  and 
facilities. 

•  The  Federal  Bureau  of  Investigation,  the  National  Ground 
Intelligence  Center,  Department  of  Homeland  Security, 


and  other  Federal  agencies  interface  with  ABIS  to  identify 
biometrics  matches  in  support  of  U.S.  criminal  cases,  border 
control,  and  intelligence  watchlists,  respectively. 

Major  Contractor 

Northrop  Grumman,  Information  Technology 
(NGIT)  -  Fairmont,  West  Virginia 


Activity 

•  ABIS  was  first  developed  as  a  prototype  in  2004  in  response 
to  a  Joint  Urgent  Operational  Need  Statement.  ABIS  1.0  was 
deployed  to  BIMA  in  January  2009  as  a  prototype  system  to 
provide  multi-modal  and  multifunctional  biometric  capabilities 
to  assist  in  the  Global  War  on  Terrorism  and  subsequently  in 
Overseas  Contingency  Operations. 

•  Since  2004,  DOT&E  designated  all  biometrics  programs  be 
placed  on  the  T&E  oversight  list  as  pre-Major  Automated 
Information  Systems.  As  such,  although  not  a  formal  program 
of  record,  ABIS  is  included  on  DOT&E  oversight. 

•  In  J anuary  20 1 1 ,  U SD(AT &L)  issued  an  Acquisition  Decision 
Memorandum  establishing  ABIS  1.2  as  the  baseline  for  BEC  0 
upon  completion  of  a  Full  Deployment  Decision  (originally 
scheduled  for  FY11). 

•  Between  December  2012  and  June  2013,  the  PMO  conducted 
a  number  of  customer  (developmental)  tests  to  determine  if 
ABIS  1.2  enabled  the  operators  to  access  the  functions  they 
needed  to  perform  their  duties  and  if  the  system  would  react 
with  consistent,  accurate,  and  useful  reports,  displays,  or  other 
responses. 

•  In  August  2013,  the  PMO  deployed  ABIS  1.2  as  the  system  of 
record  directly  supporting  real-world  operations  for  10  days. 
During  the  August  2013  deployment,  U.S.  Special  Operations 
Command  (USSOCOM)  documented  31  high-priority 
deficiencies  and  U.S.  Central  Command  (USCENTCOM) 
documented  1 1  high-priority  deficiencies  that  affected  mission 
accomplishment  due  to  deficiencies  in  the  ABIS  1.2  baseline 
affecting  the  effectiveness  and  suitability  of  the  system. 
Following  the  August  2013  deployment,  senior-level  user 
representatives  from  both  USSOCOM  and  USCENTCOM 
issued  memoranda  requesting  that  formal  operational  testing 
be  conducted  on  future  ABIS  upgrades  prior  to  deploying  the 
upgrades,  to  help  prevent  further  deployments  that  negatively 
affect  missions. 

•  In  February  and  March  2014,  ATEC  performed  a  customer 
test.  The  test  operated  at  multiple  sites  with  the  primary  site 
being  BIMA.  The  customer  test  used  a  variety  of  recorded 
data  submissions  that  were  modified  to  allow  submission  to 
operational  handheld  biometric  devices  and  sample  stored 
submissions  selected  or  designed  to  cause  the  system  to 
perform  reviews  and  produce  responses  in  accordance  with  the 
user  cases  related  to  the  specific  problems  under  test. 


•  ATEC  performed  a  two-phased  operational  test  on  ABIS  1.2. 
The  first  phase  of  the  operational  test  was  conducted 
August  7  -  28,  2014.  The  second  phase  was  conducted 
October  17-22,  2014.  The  lack  of  proper  Program  Office 
test  planning  resulted  in  a  compressed  operational  test  plan 
review  and  approval  timeline.  ATEC  submitted  the  test  plan 
for  DOT&E  approval  on  August  1 ,  just  six  days  prior  to  the 
planned  test  start  date.  To  correct  key  test  plan  shortcomings, 
on  August  5,  2014,  DOT&E  provided  critical  comments  to 
ensure  test  adequacy.  DOT&E  formally  approved  ATEC ’s 
test  plan  on  August  12,  2014,  in  time  to  support  Phase  1  of  the 
operational  test,  which  the  Army  decided  to  begin  on  August  7. 
Due  to  the  compressed  timeline,  Phase  1  of  the  operational  test 
was  allowed  to  proceed  even  though  an  adequate  test  plan  to 
address  Phase  2  had  not  been  approved  by  DOT&E. 

•  The  second  phase  of  the  test,  which  was  supposed  to  begin 
directly  following  the  first  phase,  was  delayed  to  address 
problems  discovered  during  the  first  phase  of  testing. 

Assessment 

•  During  the  August  2013  deployment,  testing  revealed  that  the 
interfaces  between  the  current  1 .0  system  and  its  customers 
are  not  fully  defined  and  documented.  Interfaces  have  been 
created  and  sustained  on  an  ad-hoc  basis  by  BIMA  in  support 
of  mission  needs.  Documentation  of  the  interfaces  and 
services  required  by  ABIS  1.2  has  required  close  collaboration 
between  operators  and  the  system  engineers  responsible 

for  the  1.0  and  1.2  systems.  The  Joint  Interoperability  Test 
Command  is  tasked  with  verification  of  interoperability  of 
ABIS  1.2  and  testing  is  scheduled  to  be  conducted  from 
November  3  -  14,  2014. 

•  During  the  ATEC  customer  test  performed  in  February  and 
March  2014,  ABIS  1.2  operated  throughout  the  period  with 
no  significant  system  disruptions.  The  customer  test  was 
conducted  in  a  non-operational  environment  in  which  data 
submissions  were  made  using  previously  recorded  submission 
data  whose  flow  can  be  controlled  by  the  system  under  test. 
The  system  processed  all  of  the  submitted  transactions  and 
ingested  all  those  transactions,  satisfying  the  processing 
specifications.  Although  there  were  some  initial  problems 
with  some  configuration  settings,  the  Watchdesk  operators, 
who  handle  customer  requests  and  monitor  submissions  and 
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responses,  or  system  administrators  were  able  to  correct 
issues  such  as  account  and/or  computer  settings,  in  order  to 
properly  receive  responses  to  submissions  from  the  users, 
correct  message  templates  are  required.  During  the  customer 
test,  some  discrepancies  were  noted  in  selected  responses 
that  required  changes  to  certain  message  templates  for  the 
responses  to  be  properly  received  as  expected. 

•  During  Phase  1  of  the  IOT&E,  the  following  problems  were 
observed: 

-  Discrepancies  between  ABIS  l.OandABIS  1.2watchlist 
hits.  ABIS  1.0  and  ABIS  1.2  were  not  fully  consistent 
in  identifying  individuals  on  the  watchlist.  Correctly 
matching  individuals  to  the  watchlist  is  a  critical  ABIS 
function.  Review  of  1 07  watchlist  hits  during  Phase  1 
found  17  watchlist  hit  discrepancies.  Further  analysis  of 
the  discrepancies  attributed  the  discrepancies  to  timing  of 
ingestion  of  daily  watchlists  between  DOD  ABIS  1.2  and 
DOD  ABIS  1.0  and  differences  between  the  contents  of  the 
daily  watchlists. 

-  Discrepancies  in  the  number  of  identities  contained  in 
the  Custom  Biometrically  Enabled  Watchlists  (BEWLs) 
generated  by  ABIS  1.0  and  ABIS  1.2.  Custom  BEWLs 
are  smaller  subsets  of  the  full  set  of  identities  contained  in 
BEWL,  which  are  used  in  the  field  to  determine  the  course 
of  action  when  a  person  of  interest  is  detained.  Custom 


BEWLs  generated  after  Phase  1  were  reviewed  and  all 
identities  provided  for  the  Custom  BEWL  were  present. 

-  Phase  1  also  demonstrated  ABIS  1.2  problems  that  (1) 
negatively  affected  successful  completion  of  Latent 
and  Biometric  examination  workflows,  (2)  prevented  a 
significant  amount  of  data  sharing  with  the  Federal  Bureau 
of  Investigation  upon  deployment,  and  (3)  affected  the 
ability  of  the  Watchdesk  and  Examiners  from  effectively 
completing  some  tasks.  During  Phase  2,  the  software 
patches  and  changes  in  standard  operating  procedures 
resolved  the  problems  noted  in  these  areas  during  Phase  1. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  PMO  has  not 

adequately  addressed  all  of  the  previous  recommendations. 

The  PMO  still  needs  to: 

1 .  Conduct  a  baseline  assessment,  to  include  the  definition  of 
external  interfaces  to  the  current  system  and  customers. 

2.  Institutionalize  a  formal  standards  conformance  program, 
listing  external  systems  that  have  been  independently 
verified  to  be  interoperable  with  the  biometrics  enterprise. 

•  FY 14  Recommendation. 

1 .  The  Army  should  resolve  cybersecurity  findings  from  the 
IOT&E  Red  Team  assessment  and  complete  an  adversarial 
assessment  of  ABIS  1.2  during  FOT&E. 
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Excalibur  Increment  1b  M982E1 


Executive  Summary 

•  The  Army  conducted  the  Excalibur  Increment  lb  IOT&E 
at  Yuma  Proving  Ground,  Arizona,  in  January  and 
February  2014.  DOT&E  published  the  combined 
IOT&E/LFT&E  report  on  June  18,  2014. 

•  The  Excalibur  Increment  lb  projectile  is  operationally 
effective.  Relative  to  standard  projectiles,  its  accuracy  and 
lethality  allow  cannon  artillery  units  to  effectively  engage 
more  point  targets  with  better  effects  using  fewer  projectiles 
in  complex  urban  terrain. 

•  The  Excalibur  Increment  lb  is  operationally  suitable. 
Excalibur  Increment  lb  met  its  90  percent  reliability 
requirement  by  demonstrating  a  system  reliability  point 
estimate  of  97  percent  in  the  IOT. 

•  The  Army  awarded  a  full-rate  production  contract  for 
757  projectiles  on  June  27,  2014. 

System 

•  Excalibur  Increment  lb  is  a  precision-guided, 
extended-range,  155-millimeter  unitary,  high-explosive 
artillery  projectile.  It  is  fin-stabilized  and  glides  to  a  target. 

•  Excalibur  uses  GPS  and  an  Inertial  Measurement  Unit  to 
attack  point  targets  with  accuracy  of  less  than  10  meters 
from  the  desired  aim  point  (in  an  unjammed  environment). 

•  The  Army  developed  the  High-Explosive,  Unitary  (Block  I) 
projectile  in  three  spirals  of  increasing  capability  (la-1,  la-2, 
and  lb).  The  la-1  projectiles  use  aerodynamic  lift  generated 
by  canards  to  extend  range  out  to  24  kilometers  without  the 
maximum  propellant  charge.  The  la-2  and  lb  projectiles 
add  base  bleed  technology  and  use  of  the  maximum 
propellant  charge  to  further  increase  range  to  beyond 


35  kilometers.  Increment  lb  projectiles  improve  reliability 
and  reduce  cost. 

Mission 

Field  Artillery  units  use  Excalibur: 

•  To  attack  enemy  targets  in  support  of  maneuver  operations 
at  a  greater  range  and  with  increased  accuracy  than  standard 
high-explosive  munitions. 

•  To  support  the  close  fight  in  urban  and  complex  environments, 
striking  critical  targets  that  must  be  engaged  at  extended 
ranges  or  in  areas  where  minimal  collateral  damage  is  desired. 

•  To  support  fire  missions  against  personnel  and  point  targets 
such  as  threat  forces  emplacing  IEDs,  light  material,  and 
personnel  within  structures. 

Major  Contractor 

Raytheon  Missile  Systems  -  Tucson,  Arizona 


Activity 

•  In  December  2013,  the  Army  conducted  the  Excalibur 
Increment  lb  First  Article  Test  with  low-rate  initial  production 
projectiles. 

•  The  Army  conducted  the  Excalibur  Increment  lb  IOT&E 
at  Yuma  Proving  Ground,  Arizona,  from  January  through 
February  2014. 

•  On  June  18,  2014,  DOT&E  published  an  IOT&E/LFT&E 
report  in  support  of  the  Army’s  June  2014  Full-Rate 
Production  decision. 

•  The  Army  Acquisition  Executive  approved  full-rate 
production  for  Excalibur  Increment  lb  on  June  25,  2014. 

•  The  Army  awarded  a  full-rate  production  contract  for 
757  projectiles  on  June  27,  2014. 

•  The  Army  conducted  all  testing  in  accordance  with 
DOT&E-approved  Test  and  Evaluation  Master  Plan  and 
operational  test  plan. 


Assessment 

•  Excalibur  Increment  lb  test  plan  execution  was  adequate 
to  assess  operational  effectiveness,  suitability,  lethality,  and 
survivability. 

•  The  Excalibur  Increment  lb  projectile  is  operationally 
effective. 

-  Relative  to  standard  projectiles,  its  accuracy  and  lethality 
allow  cannon  artillery  units  to  engage  point  targets  with 
precision  effects  using  fewer  projectiles  in  complex, 
urban  terrain,  limiting  collateral  damage. 

-  DOT&E  has  reviewed  Army  combat  reporting  that 
showed  units  in  Operation  New  Dawn  and  Operation 
Enduring  Freedom  effectively  used  Excalibur  Increment 
la-1  and  la-2  projectiles  for  timely  engagement  of  targets 
in  complex  urban  environments  with  minimal  collateral 
damage. 
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-•  Excalibur  Increment  lb  achieved  required  accuracy 
irrespective  of  the  presence  or  absence  of  GPS  jamming. 
During  operational  testing,  Excalibur  Increment  lb 
demonstrated  a  median  miss  distance  of  3.7  meters  for 
projectiles  fired  at  32-kilometer  ranges  in  an  unjammed 
environment.  The  median  miss  distance  for  all 
projectiles  was  2.0  meters.  The  Army  requires  Excalibur 
Increment  lb  target  accuracy  of  10  meters. 

•  Live  fire  testing  and  analyses  concluded  the  Excalibur 
Increment  lb  projectile  is  more  lethal  against  personnel 
targets  and  light  material  targets  than  standard  high-explosive 
projectiles. 

•  The  Excalibur  Increment  lb  is  operationally  suitable.  The 
projectile  is  reliable  when  fired  at  all  ranges  up  to  its  maximum 
range  of  35  kilometers.  The  Army  requires  a  system 
reliability  of  90  percent.  During  operational  testing,  Excalibur 
Increment  lb  met  the  requirement  and  demonstrated  a  system 
reliability  estimate  of  97  percent. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed  all 
previous  recommendations. 


FY14  Recommendations.  The  Army  should  continue  to 

address  recommendations  from  DOT&E’s  June  2014  Report: 

1 .  Optimize  Advanced  Artillery  Tactical  Data  System  software 
to  employ  a  special  sheaf  with  aim  points  equally  spaced 
from  the  target  center  when  the  achieved  target  location 
error  reported  by  an  observer  is  less  than  10  meters  to 
maximize  the  effects  of  multiple  Excalibur  projectiles  fired 
against  a  single  point  target. 

2.  Update  the  Forward  Observer  System  software  so  that  fire 
support  teams  at  company  level  and  above  can  specify 
Excalibur  or  Precision  Guidance  Kit  munitions  when 
transmitting  observer-initiated  fire  mission  requests  for 
precision-guided  munitions  to  a  firing  unit.  The  current 
Forward  Observer  System  version  does  not  provide  the  option 
to  select  specific  precision  munition  shell/fuze  combinations. 

3.  Modify  current  metrics  to  better  assess  lethality  against 
different  types  of  construction  prevalent  in  theaters  of 
operation. 

4.  Collect  and  examine  data  on  the  effects  of  varied  terrain, 
vegetation,  and  body  armor  on  Excalibur  lethality  and  other 
blast-fragmentation  projectiles  with  similarly  small  fragment 
sizes. 
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Guided  Multiple  Launch  Rocket  System  -  Alternate 
Warhead  (GMLRS-AW)  XM30E1 


Executive  Summary 

•  The  M30E1  Guided  Multiple  Launch  Rocket  System  -  Alternate 
Warhead  (GMLRS-AW)  surface-to-surface  rocket  will  meet  the 
DOD  unexploded  ordnance  policy  requirements  and  replace  the 
non-compliant  GMLRS  Dual-Purpose  Improved  Conventional 
Munitions  rocket. 

•  In  F Y 1 4,  the  Army  conducted  GMLRS-AW  developmental 
testing/operational  testing  (DT/OT)  in  which  Soldier  crews 
fired  15  rockets  during  3  tactical-fire  missions  against 
threat-representative  targets. 

•  The  M30E1  GMLRS-AW  IOT&E  was  executed  from 
October  through  November  2014.  DOT&E  will  submit  a 
combined  IOT&E/LFT&E  Report  to  support  the  Army’s 
planned  March  2015  Full-Rate  Production  decision. 

System 

•  The  M30E1  GMLRS-AW  rocket  uses  Inertial  Measurement 
Unit  and  GPS  guidance  to  engage  area  targets  out  to 

70  kilometers. 

•  GMLRS-AW  uses  the  same  rocket  motor,  guidance  system, 
and  control  system  as  the  existing  M3 1 A1  GMLRS  Unitary 
warhead  rocket. 

•  The  GMLRS-AW  rockets  can  be  fired  from  the  tracked 
M270A1  Multiple  Launch  Rocket  System  and  the  wheeled 
High  Mobility  Artillery  Rocket  System  (HIMARS). 

•  The  200-pound  GMLRS-AW  high-explosive  warhead  contains 
approximately  160,000  preformed  tungsten  fragments.  This 
warhead  change  eliminates  the  unexploded  ordnance  found  in 
the  GMLRS  Dual-Purpose  Improved  Conventional  Munitions 
rockets. 

•  GMLRS-AW  meets  the  dud  rate  requirement  as  defined  by 
the  current  DOD  Policy  on  Cluster  Munitions  and  Unintended 
Harm  to  Civilians  memorandum  dated  June  19,  2008. 

•  The  procurement  objective  is  18,072  GMLRS-AW  rockets. 

The  Army  plans  to  enter  full-rate  production  in  March  2015 
following  IOT&E. 


Mission 

Commanders  will  use  GMLRS-AW  rockets  to  engage  area-  or 
imprecisely-located  targets  without  the  hazard  of  unexploded 
submunitions.  The  target  set  includes  counterfire,  air  defense, 
command  posts,  assembly  areas,  light  materiel,  and  other  high 
payoff  targets. 

Major  Contractor 

Lockheed  Martin  Missiles  and  Fire  Control  -  Dallas,  Texas 


Activity 

•  From  October  2013  to  April  20 1 4,  the  Army  completed 
the  Production  Qualification  Test  (PQT)  flight  test  series, 
which  consisted  of  5  missions  and  17  GMLRS-AW  rockets 
fired  at  short,  medium,  and  long  ranges  (16.8,  37.4,  and 
65.7  kilometers,  respectively).  PQT  missions  were  launched 
from  M270A1  launchers  and  Ml 42  HIMARS  launchers. 

Four  of  the  five  missions  were  fired  at  elements  of  targets  that 
will  be  used  in  the  IOT&E:  a  towed  howitzer  battery  including 


personnel,  a  forward  command  post,  and  a  surrogate  SA-6 
radar. 

•  During  FY 14,  the  GMLRS-AW  Program  Office  conducted 
PQT  ground  testing,  including  static  arena  tests  to  evaluate 
warhead  lethality,  temperature  shock  and  vibration  testing, 
software  development  testing,  and  system  integration  tests. 

•  In  June  2014,  Soldiers  from  the  214th  Fires  Brigade  at 
Fort  Sill,  Oklahoma,  conducted  an  integrated  DT/OT  at 
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White  Sands  Missile  Range,  New  Mexico,  consisting  of 
three,  tactical-fire  missions.  There  were  15  rockets  fired  from 
a  HIMARS  launcher. 

•  During  the  DT/OT  period,  the  Army  conducted  a 
cybersecurity  vulnerability  assessment  of  the  HIMARS  and 
M270A1  launchers  and  the  Advanced  Field  Artillery  Tactical 
Data  System. 

•  The  GMLRS-AW  IOT&E  was  executed  from  October  through 
November  2014. 

•  DOT&E  will  submit  a  combined  IOT&E/LFT&E  report  to 
support  the  Army’s  planned  March  2015  Full-Rate  Production 
decision. 

•  The  Army  conducted  the  PQT  and  DT/OT  in  accordance  with 
the  DOT&E-approved  Test  and  Evaluation  Master  Plan  and 
test  plans. 

Assessment 

•  GMLRS-AW  munition  is  accurate  and  reliable.  DOT&E  is 
evaluating  the  effectiveness  and  lethality  of  the  GMLRS-AW 
munition.  GMLRS-AW  munition  contains  no  submunitions 
minimizing  the  unintended  harm  to  civilians  and  infrastructure 
associated  with  unexploded  ordnance  from  cluster  munitions. 


•  The  PQT  demonstrated  the  GMLRS-AW  rocket  is  reliable 
(17  successes  in  17  flights)  and  accurate  (median  miss 
distance  of  2.1  meters).  Reliability  and  accuracy  were  further 
demonstrated  in  the  June  2014  DT/OT  (15  successes  in 

15  flights,  and  median  miss  distance  of  2.7  meters).  There 
is  no  system  requirement  for  accuracy.  The  contractor 
specification  is  less  than  15  meters  Circular  Error  Probable. 

•  During  DT/OT,  Soldier  crews  were  able  to  process  and  execute 
GMLRS-AW  fire  missions  using  the  HIMARS  launcher  and 
the  Advanced  Field  Artillery  Tactical  Data  System  command 
and  control  software. 

•  The  GMLRS-AW  system  met  the  requirement  for  the  three  fire 
missions  in  the  DT/OT,  including  one  fire  mission  where  GPS 
jamming  occurred. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed  all 
previous  recommendations. 

•  FY 14  Recommendations.  None. 
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Integrated  Personnel  and  Pay  System  -  Army  (IPPS-A) 


Executive  Summary 

•  Integrated  Personnel  and  Pay  System  -  Army  (IPPS-A) 
is  a  two  increment  program  that  streamlines  Army 
Human  Resources  processes  and  enhances  the  efficiency 
and  accuracy  of  Army  personnel  and  pay  procedures 

to  support  Soldiers  and  their  families.  Through  a  three 
phased-delivery  approach.  Increment  1  of  the  IPPS-A 
program  provides  the  foundational  data  for  a  single, 
integrated  military  personnel  and  pay  system  for  all  three 
Army  components:  the  active-duty  Army,  the  Army 
Reserve  National  Guard  (ARNG),  and  the  Army  Reserve. 

•  The  Army  Test  and  Evaluation  Command 
(ATEC)  and  ARNG  conducted  an  IOT&E  event 
on  February  19  -  2 1,  2014,  at  die  ARNG  Bureau 
in  Aldington,  Virginia,  in  accordance  with  an 
ATEC-approved  test  plan.  IPPS-A,  as  it  exists  in 
Increment  1,  is  effective  and  survivable.  A  suitability 
assessment  is  deferred  until  Increment  2.  The  capabilities 
available  in  this  increment  are  limited;  the  program  should 
continue  to  improve  IPPS-A  in  order  to  deliver  die  foil  set 
of  necessary  capabilities. 

•  The  IPPS-A  Increment  1  system  demonstrated  the 
capability  to  produce  its  primary  product,  a  Soldier’s 
Record  Brief  (SRB),  which  is  a  single,  integrated 
compilation  of  personnel  and  pay  data  collected  from 
various,  external  audioritative  sources. 

System 

•  EPPS-A  is  a  two  increment  program  that  streamlines  Army 
Human  Resources  processes  and  enhances  the  efficiency 
and  accuracy  of  Army  personnel  and  pay  procedures  to 
support  Soldiers  and  their  families.  Increment  1  interfaces 
with  legacy  applications  to  create  a  dusted,  foundational 
database.  All  authoritative  data  remain  in  the  legacy 
systems  for  Increment  1 .  Increment  2  will  become  the 
audioritative  data  source  as  the  necessary  functionality  of 
the  legacy  systems  to  be  subsumed  is  incorporated. 

•  It  is  a  web-based  tool,  available  24  hours  a  day,  accessible 
to  Soldiers,  Human  Resources  professionals.  Combatant 
Commanders,  personnel  and  pay  managers,  and  other 
authorized  users  throughout  the  Army.  IPPS-A  improves 
die  delivery  of  military  personnel  and  pay  services  and  also 
provides  internal  controls  and  audit  procedures  to  prevent 
erroneous  payments  and  loss  of  funds. 

•  IPPS-A  interfaces  with  15  other  Army  and  DOD  systems 
to  acquire  personnel  and  pay  data,  which  it  integrates 
into  a  single  record  for  each  Soldier.  These  systems 
include  the  Defense  Enrollment  Eligibility  Reporting 
System  -  Personnel  Data  Repository,  Electronic  Military 
Personnel  Office,  Standard  Installation  and  Division 


Personnel  Reporting  System  -  Guard,  and  Total  Army 
Personnel  Data  Base  -  Reserve.  IPPS-A’s  SRB  acts  as  a 
trusted,  but  non-autlioritative.  display  of  data  contained  in 
the  various  external  systems;  any  changes  required  to  the 
data  must  be  made  within  the  existing  15  Army  and  DOD 
Personnel  systems  and  cannot  be  accomplished  within 
IPPS-A.  The  SRB  displays  a  Soldier’s  military  career 
personal  information,  qualification  skills,  training,  assignment 
history,  and  various  other  Soldier  attributes. 


Soldiers  will  use  IPPS-A  as  a  single,  integrated  personnel  and 
pay  system  that  will  provide  personnel  and  pay  management 
functionality  for  all  Army  Components.  Army  Components  will 
use  IPPS-A  to  manage  their  members  across  the  full  operational 
spectrum  dming  peacetime,  war.  through  mobilization  and 
demobilization,  capturing  timely  and  accinate  data  throughout. 
Additionally,  Commanders  will  possess  a  comprehensive  system 
for  accountability  and  information  to  support  command  decisions 
regardless  of  component  or  geographic  location. 


Major  Contractor 

EDC  Consulting  LLC  -  McLean,  Virginia 


Mission 


IPPS-A 
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Activity 

•  ATEC  conducted  a  risk  assessment  of  Increment  1  on 
June  19,  2014,  in  accordance  with  DOT&E  Information 
and  Business  Systems  Policy.  Due  to  the  low  risk  of 
the  capabilities  being  delivered  in  Increment  1,  the  risk 
assessment  allowed  for  the  delegation  of  test  plan  approval 
to  ATEC. 

•  ATEC  and  ARNG  executed  the  IOT&E  event  on 
February  19  -  21,  2014,  at  the  ARNG  Bureau  in  Arlington, 
Virginia. 

-  The  three-day  event  consisted  of  user  training  on  SRB 
access,  observations  of  user  utilization  oflPPS-Ato  view 
their  SRB,  and  user  response  to  the  survey  questions. 

-  One  hundred  and  ninety  ARNG  Soldiers  received  training 
on  February  19,  2014,  in  order  to  login  using  their 
Common  Access  Cards,  view  their  SRB,  and  complete  an 
online  survey  and  print  out  their  SRB. 

•  Prior  to  the  IOT&E  event  on  February  12  -  13,  2014,  the 
Threat  Systems  Management  Office  conducted  cybersecurity 
testing  on  IPPS-A  in  accordance  with  the  DOT&E 
Information  Assurance  policy. 

Assessment 

•  IPPS-A  Increment  1,  as  delivered,  provides  an  SRB  that  is 
viewable  through  a  web  interface  and  can  be  printed  out. 
Increment  1  does  not  provide  the  capability  to  add  or  edit 
personnel  data.  The  ability  to  edit  personnel  and  pay  data  will 
be  phased  in  during  the  four  releases  in  Increment  2. 

•  The  results  of  the  online  survey  indicate  the  system  was  easy 
to  use  and  the  resources  necessary  to  obtain  and  interpret  the 
data  on  the  SRB  were  adequate.  The  results  also  indicate 
that  the  training  received  and  online  resources  available  were 
sufficient  for  most  Soldiers.  Very  few  of  the  Soldiers  used  the 
help  desk,  and  the  associated  survey  results  did  not  provide  a 
significant  response  as  to  whether  they  were  satisfied  with  the 
help  desk  support. 

•  Out  of  190  participants  surveyed  during  the  IOT&E, 

181  (95.2  percent)  reported  errors  in  their  SRB. 
Developmental  testing  verified  that  IPPS-A  accurately  ingests, 
processes,  and  displays  personnel  data  in  the  SRB.  Therefore, 
any  data  errors  within  the  SRB  discovered  during  this  test 


reflect  incorrect  data  received  from  the  external,  legacy 
systems. 

•  The  Army  is  working  to  correct  the  errors  in  the  various 
databases,  which  feed  IPPS-A.  The  Army  G-l  is  tracking  the 
data  correction  process  and  intends  to  provide  a  report  prior  to 
the  end  of  fielding  IPPS-A  Increment  1 . 

•  The  SRB  can  be  categorized  into  1 1  sections  and  the  Header 
and  Footer  sections.  Participants  found  data  errors  in  all 

11  sections  and  the  Header  and  Footer  sections  of  the  SRB. 
Sections  where  more  than  50  percent  of  the  participants  had 
data  errors  include  Personal/family  data,  Civilian  Education, 
and  Military  Education.  Preliminary  analysis  shows  the 
legacy  sources  contributing  to  most  errors  are  Army  Training 
Requirements  and  Resources  System,  Reserve  Component 
Manpower  System,  and  Standard  Installation  and  Division 
Personnel  Reporting  System. 

•  The  results  of  the  Cyber  Vulnerability  testing  found 

6  Category  1  and  86  Category  2  deficiencies.  Cybersecurity 
results  from  a  subsequent  verification  of  fixes  event  indicate 
that  all  major  vulnerabilities  discovered  during  the  IOT&E 
have  been  mitigated. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations. 

1 .  The  Army  should  continue  to  track  satisfaction  with 
help  desk  procedures  and  support  to  determine  actual 
satisfaction  with  help  desk  services  provided  to  Soldiers  to 
support  successful  fielding  of  Increment  1  to  the  active-duty 
Army  and  Army  reserve. 

2.  To  ensure  successful  development  and  fielding  of 
Increment  2,  the  Army  should: 

■  Continue  to  pursue  correction  of  personnel  data  in 
the  1 5  Army  and  DOD  systems  that  provide  the  data 
necessary  to  create  correct  SRBs. 

■  Confirm  the  data  verification  process  successfully 
updates  records  in  external  systems,  which  will  enable 
IPPS-A  to  display  an  accurate  SRB. 
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Joint  Battle  Command  -  Platform  (JBC-P) 


Executive  Summary 

•  DOT&E  released  a  November  2013  IOT&E  report  that 
assessed  Joint  Battle  Command  -  Platform  (JBC-P)  software 
build  5.0  as  operationally  effective,  not  operationally  suitable, 
and  not  survivable.  This  report  was  based  upon  the  May  2013 
JBC-P  software  build  5.0  IOT&E. 

•  In  November  2013,  the  Army  approved  a  fielding  decision  for 
JBC-P  software  build  5.1  based  upon  the  May  2013  JBC-P 
IOT&E  and  demonstrated  corrections  to  discovered 
deficiencies. 

•  In  May  2014,  the  Army  and  Marine  Corps  conducted  a 
JBC-P  software  build  6.0  Multi-Service  Operational  Test  and 
Evaluation  (MOT&E)  to  support  fielding  decisions  in  both 
Services.  While  software  build  6.0  delivered  several  enhanced 
capabilities,  it  introduced  deficiencies  which  significantly 
detracted  from  mission  capabilities  and  led  to  a  change  in  the 
assessment.  DOT&E  assessed  JBC-P  software  build  6.0  as: 

-  Not  operationally  effective  due  to  low  message 
completion  rates,  phantom  Mayday  messages,  inaccurate 
representation  of  blue  force  icons,  and  the  poor 
performance  of  JBC-P  Logistics  (Log) 

-  Not  operationally  suitable  due  to  reliability  that  was  below 
the  Army’s  requirement  for  five  of  seven  JBC-P  hardware 
variants,  deficiencies  in  training  provided  to  Soldiers,  and 
lack  of  a  force  structure  to  support  JBC-P  Log 

-  Not  survivable  due  to  cybersecurity  vulnerabilities 

System 

•  JBC-P  is  a  networked  battle  command  information  system 
that  enables  units  to  share  near  real-time  friendly  and  enemy 
situational  awareness  information,  operational  maps  and 
graphics,  and  command  and  control  (C2)  messages.  The  Army 
and  Marine  Corps  intend  JBC-P  to  achieve  platform-level 
interoperability  for  ground  vehicles,  dismounted 
Soldiers/Marines,  and  aviation  assets  operating  in  land/littoral 
and  joint  operational  environments. 

•  JBC-P  is  an  upgrade  to  the  Force  XXI  Battle  Command 
Brigade  and  Below  Joint  Capabilities  Release  and  provides  the 
following  improvements: 

-  Tactical  chat  combined  with  chat  room  capability, 
providing  enhanced  collaboration  for  commanders 

-  Improved  mission  command  applications  for  planning  and 
execution 

-  A  more  intuitive  graphical  user  interface  with  improved 
display  of  maps  and  images 

-  Enhanced  blue  force  situational  awareness  between  mobile 
platforms,  Tactical  Operational  Centers,  and  dismounted 
Soldiers  equipped  with  Nett  Warrior 

-  JBC-P  Log  provides  tracking  of  logistics  cargo  within 
the  unit’s  area  of  operations  through  the  use  of  Radio 
Frequency  Identification  tags 


2  -  JBC-P  on  Joint  Version  -  5  (JV-5)  Block  2 

3  -  JBC-P  Screen  Shot 

4  -  JBC-P  LOG  on  Military  Rugged  Tablet  Plus  (MRT+) 
5 -JBC-P  on  JV-5  Block  2 


-  Hybrid  capability  to  connect  JBC-P  across  different 
networks  through  the  use  of  its  Network  Services  Gateway 
and  associated  terrestrial  and  satellite  radios 

•  JBC-P  is  fielded  in  both  mobile  and  command  post  versions. 

JBC-P  communications  is  supported  by: 

-  Blue  Force  Tracker  2  satellite  communications  for  mobile 
operations 

-  Tactical  radios  for  connectivity  between  JBC-P-equipped 
vehicles  and  to  support  dismounted  operations 

-  Tactical  Internet  for  command  post  operations 

Mission 

Army,  Marine  Corps,  and  Special  Operations  Forces  commanders 
use  JBC-P  to  provide  integrated,  on-the-move,  near  real-time 
battle  command  information  and  situational  awareness  from 
brigade  to  maneuver  platform  to  dismounted  Soldiers/Marines. 

Major  Contractor 

Software  Engineering  Directorate,  U.S.  Army  Aviation 
&  Missile  Research,  Development  &  Engineering 
Center  -  Huntsville,  Alabama 
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Activity 

•  During  the  October  2013  Network  Integration  Evaluation  14.1, 
the  Army  conducted  a  JBC-P  software  build  5.1  customer  test 
to  demonstrate  correction  of  the  May  2013  JBC-P  IOT&E 
software  build  5.0  deficiencies. 

•  In  November  2013,  DOT&E  released  a  JBC-P  IOT&E  report 
to  support  the  Army’s  JBC-P  software  5.1  fielding  decision. 

•  In  November  2013,  the  Army  completed  a  fielding  decision  for 
JBC-P  software  build  5.1,  based  upon  the  May  2013  IOT&E 
and  correction  of  noted  deficiencies.  The  fielding  decision 
was  contingent  upon  completion  of  Army  Interoperability 
Certification. 

•  During  FY14,  the  program  received  a  conditional  material 
release,  and  completed  Army  and  Joint  Interoperability 
Certifications  on  JBC-P  software  build  5.1. 

•  During  the  May  2014  Network  Integration  Evaluation  14.2, 
the  Army  and  Marine  Corps  conducted  a  JBC-P  software 
build  6.0  MOT&E  to  support  fielding  decisions  for 

both  Services.  The  test  was  conducted  according  to  a 
DOT&E-approved  Test  and  Evaluation  Master  Plan  and 
test  plan,  and  employed  an  Army  brigade  with  an  attached 
Marine  Corps  battalion  conducting  missions  under 
operationally  realistic  conditions. 

•  DOT&E  will  publish  an  MOT&E  report  in  FY15. 

Assessment 

•  Based  on  results  from  the  2013  JBC-P  software  build  5.0 
IOT&E,  DOT&E  released  a  JBC-P  IOT&E  report  in 
November  2013,  which  assessed  JBC-P  as: 

-  Operationally  effective  in  supporting  Army  commanders 
and  Soldiers  with  situational  awareness,  C2  messages,  and 
chat  when  operating  from  Tactical  Operational  Centers 
and  on-the-move  in  tactical  vehicles.  JBC-P  served  as  the 
Soldiers’  primary  tool  for  C2  when  on-the-move. 

-  Not  operationally  suitable  due  to  poor  reliability  (less 
than  the  Army’s  reduced  requirement)  and  deficiencies  in 
training  provided  to  Soldiers. 

-  Not  survivable  due  to  Information  Assurance 
vulnerabilities. 

•  Based  upon  MOT&E,  DOT&E  assessed  the  effectiveness  of 
the  JBC-P  software  build  6.0  for  combat  operations.  While 
software  build  6.0  delivered  several  enhanced  capabilities, 

it  introduced  deficiencies  which  significantly  detracted  from 
mission  capabilities  and  led  to  an  assessment  that  the  JBC-P 
was  now  not  effective.  During  MOT&E,  JBC-P: 

-  Demonstrated  the  ability  to  pass  situational  awareness 
messages. 

-  Provided  effective  chat  communications  between  all 
echelons  of  the  brigade. 

-  Did  not  meet  message  completion  and  timeliness 
requirements  for  C2  and  survivability  data. 


-  Generated  numerous  false  Mayday  messages  and  provided 
inaccurate  representations  of  blue  force  icons,  which 
reduced  the  Soldiers’  confidence  in  the  system. 

-  Did  not  provide  an  effective  means  to  track  logistics 
cargoes  using  JBC-P  Log. 

•  Based  upon  MOT&E,  DOT&E  assessed  JBC-P  as  not 
operationally  suitable  and  highlighted  the  following 
deficiencies: 

-  The  majority  of  JBC-P  hardware  did  not  meet  the  Mean 
Time  Between  Essential  Function  Failure  reliability 
requirement  of  290  hours.  Two  of  the  seven  JBC-P 
hardware  variants  met  their  Mean  Time  Between  Essential 
Function  Failure  reliability  requirement,  the  JV-5 

(469  hours  lower  confidence  bound)  and  JV-5  Block  II 
(895  hours  lower  confidence  bound). 

-  Soldiers  experienced  problems  with  spontaneous  reboots 
and  shared  user  displays  within  Warfighter  Information 
Network  -  Tactical-equipped  vehicles. 

-  Although  improved  since  IOT&E,  training  does  not  afford 
Soldiers  and  leaders  the  ability  to  use  all  of  the  features 
of  JBC-P.  Soldiers  require  more  hands-on  training  and 
leaders  require  extended  leader  training. 

-  Logistics  units  did  not  have  signal  Soldiers  required 
to  configure,  operate,  and  maintain  JBC-P  Log.  Units 
diverted  Soldiers  from  other  unit  missions  (e.g.  fuel 
handler)  to  perform  this  duty. 

•  The  JBC-P  MOT&E  demonstrated  the  system  as  not 
survivable  against  threat  computer  network  operations. 

While  improved  compared  to  IOT&E  results,  the  Anny  needs 
to  further  improve  JBC-P’s  cybersecurity. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  made 
improvements  in  deficiencies  noted  during  IOT&E,  yet 
still  needs  to  improve  JBC-P  reliability,  training,  and 
cybersecurity. 

•  FY 14  Recommendations.  The  Army  should: 

1.  Improve  JBC-P  effectiveness  by  correcting  MOT&E 
deficiencies  to  include  low  message  completion  rates, 
phantom  Mayday  messages,  inaccurate  blue  force  icon 
representation,  and  poor  JBC-P  Log  performance. 

2.  Improve  JBC-P  reliability  by  improving  hardware  variants 
that  did  not  meet  requirements  and  correcting  software 
build  6.0  deficiencies. 

3.  Assess  Army  force  structure  to  support  JBC-P  and  JBC-P 
Log,  and  provide  necessary  Soldiers  to  configure,  operate, 
and  maintain  the  system. 

4.  Improve  JBC-P  leader  and  Soldier  training. 

5.  Correct  cybersecurity  survivability  deficiencies 
demonstrated  during  the  JBC-P  MOT&E. 
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Joint  Light  Tactical  Vehicle  (JLTV) 
Family  of  Vehicles  (FoV) 


Executive  Summary 

•  Army  and  Marine  Corps  units  equipped  with  Joint  Light 
Tactical  Vehicles  (JLTVs)  with  inherent  armor  demonstrated 
the  ability  to  execute  air  assault  missions. 

•  Based  on  developmental  test/operational  test  (DT/OT), 

Marine  Corps  units  equipped  with  JLTVs  demonstrated  the 
ability  to  execute  amphibious  assault  missions. 

•  The  JLTVs  have  large  visual  signature  and  their 
slow  maneuver  time  from  ship  to  shore  prevents  a 
Marine  Expeditionary  Unit  from  executing  assault  missions 
with  tactical  surprise,  increases  the  time  to  close  combat 
power  ashore,  and  renders  the  unit  vulnerable  to  threats. 
Testing  showed  that  JLTVs  are  slower  to  load,  prepare  for 
fording,  and  transition  to  maneuver  ashore  than  HMMWV  due 
to  their  larger  vehicle  size,  delays  that  occur  while  awaiting 
suspension  mode,  and  other  vehicle  adjustments  (e.g.,  tire 
pressure). 

•  In  September  20 1 4,  the  Army  and  Marines  conducted  the 
JLTV  Family  of  Vehicles  (FoV)  Limited  User  Test  (LUT)  at 
Fort  Stewart,  Georgia. 

•  From  November  2013  through  October  2014,  the  Army 
conducted  Engineering  Manufacturing  and  Development 
(EMD)  system-level  ballistic  testing  of  the  JLTV  designs. 

•  The  program  will  begin  Source  Selection  Board  activities 
to  down  select  to  a  single  contractor  in  January  2015  and  is 
expected  to  be  completed  by  July  2015. 

•  The  Milestone  C  Low-Rate  Initial  Production  decision  is 
planned  for  July  2015. 

•  The  Army  conducted  all  testing  in  accordance  with  a 
DOT&E-approved  test  plan  and  Test  and  Evaluation  Master 
Plan  (TEMP). 

System 

•  The  JLTV  FoV  is  the  Marine  Corps  and  Army  partial 
replacement  for  the  High  Mobility  Multi-purpose  Wheeled 
Vehicle  (HMMWV)  fleet.  The  Services  intend  the  JLTV  to 
provide  increased  crew  protection  against  IED  and  underbody 
attacks,  improved  mobility,  and  higher  reliability  than  the 
HMMWV. 

•  The  JLTV  FoV  consists  of  two  vehicle  categories:  the  JLTV 
Combat  Tactical  Vehicle,  designed  to  seat  four  passengers; 
and  the  JLTV  Combat  Support  Vehicle,  designed  to  seat  two 
passengers. 

•  The  JLTV  Combat  Tactical  Vehicle  has  a  3,500-pound  payload 
and  3  mission  package  configurations: 

-  Close  Combat  Weapons  Carrier  Vehicle 

-  General  Purpose  Vehicle 

-  Heavy  Guns  Carrier  Vehicle 


Lockheed  Martin 
Systems  JLTV 


Oshkosh  Corporation  JLTV 


•  The  JLTV  Combat  Support  Vehicle  has  a  5,100-pound  payload 
and  2  configurations: 

-  Utility  Prime  Mover 

-  Shelter  Carrier 

•  The  JLTV  program  is  using  a  competitive  prototype 
acquisition  strategy.  During  the  EMD  phase,  the  program 
tested  vehicles  from  three  contractors. 
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Mission 

•  Military  units  employ  JLTV  as  a  light,  tactical-wheeled 
vehicle  to  support  all  types  of  military  operations.  JLTVs  are 
used  by  airborne,  air  assault,  amphibious,  light,  Stryker,  and 
heavy  forces  as  reconnaissance,  maneuver,  and  maneuver 
sustainment  platforms. 

•  Small  ground  combat  units  will  employ  JLTV  in  combat 
patrols,  raids,  long-range  reconnaissance,  and  convoy  escort. 


Major  Contractors 

•  AM  General  -  South  Bend,  Indiana 

•  Lockheed  Martin  Corporation  -  Grand  Prairie,  Texas 

•  Oshkosh  Corporation  -  Oshkosh,  Wisconsin 


Activity 

•  In  April  2014,  the  Army  and  Marine  Corps  units  executed 
air  assault  missions  during  DT/OT  at  Aberdeen  Proving 
Ground,  Maryland,  using  CH-47F  and  CH-53E  helicopters. 
The  Marine  Corps  unit  executed  amphibious  assault  missions 
at  Joint  Expeditionary  Base  Little  Creek,  Fort  Story,  Virginia, 
using  Landing  Craft  Utility  ships. 

•  In  July  2014,  the  Army  Test  and  Evaluation  Command  (ATEC) 
conducted  reliability,  availability,  and  maintainability  testing 
on  all  three  vendors’  JLTVs  at  Aberdeen  Proving  Ground  and 
Yuma  Proving  Ground,  Arizona.  The  objective  of  testing  was 
to  discover  failure  modes,  implement  corrective  actions,  and 
assess  whether  the  vendors’  vehicles  could  meet  the  required 
Mean  Miles  Between  Hardware  Mission  Failure  (MMBHMF) 
prior  to  the  Milestone  C  decision. 

•  In  August  2014,  the  ATEC  completed  automotive  testing  on 
the  vendors’  JLTVs. 

•  The  Marine  Corps  demonstrated  its  automotive  performance 
requirements  when  outfitted  with  the  Army’s  higher  level 
of  underbody  protection  during  testing.  The  Marine 
Corps  removed  their  separate,  lower  underbody  protection 
requirement.  In  the  future  production  and  deployment  phase, 
all  vehicles  tested  will  be  armored  to  meet  a  single  set  of 
under-vehicle  protection  requirements. 

•  The  program  began  developing  the  JLTV  FoV  Milestone  C 
TEMP  to  reflect  the  T&E  activities  for  the  production  and 
deployment  phase  in  June  2014. 

•  In  September  2014,  the  Army  and  Marines  conducted  the 
JLTV  LUT  at  Fort  Stewart,  Georgia.  When  the  LUT  ended 
in  November  2014,  the  Army  test  unit  had  completed 
three,  96-hour  scenarios  and  the  Marine  Corps  test  unit  had 
completed  one,  96-hour  scenario  at  an  operational  tempo 
consistent  with  the  JLTV  Operational  Mode  Summary /Mission 
Profile.  Analysis  of  the  JLTV  LUT  data  is  ongoing. 

•  The  program  will  begin  Source  Selection  Board  activities 
to  down  select  to  a  single  contractor  in  January  2015  and  is 
expected  to  be  completed  by  July  2015. 

•  ATEC  completed  the  EMD  phase  of  system-level  live 
fire  testing  (50  tests  total  across  the  three  vendors)  in 
October  2014. 

•  The  Milestone  C  Low-Rate  Initial  Production  decision  is 
planned  for  July  2015. 

•  The  Army  conducted  all  testing  in  accordance  with  a 
DOT&E-approved  test  plan  and  TEMP. 


Assessment 

•  Vendors’  JLTVs  demonstrated  between  528  and 

2,194  MMBHMF  during  reliability  growth  testing.  The 
vehicles  were  required  to  demonstrate  3,196  MMBHMF  prior 
to  the  JLTV  LUT. 

•  Based  on  the  DT/OT,  Army  and  Marine  Corps  units  equipped 
with  the  JLTVs  with  inherent  armor  can  execute  air  assault 
missions.  The  three  JLTV  contractor  vehicles  were  more 
difficult  to  rig,  de-rig,  and  load  weapons  due  to  vehicle  height 
and  lack  of  vehicle  handholds  and  footholds  than  HMMWV. 
They  had  limited  space  to  carry  crew,  mission  essential 
equipment,  weapons,  and  their  sustainment  load  because  of 
the  small  interior  compartment. 

•  Marine  Corps  units  equipped  with  the  JLTVs  demonstrated 
the  ability  to  execute  amphibious  assault  missions  during 
DT/OT. 

•  The  JLTVs  have  large  visual  signature,  and  their  slow 
maneuver  time  from  ship  to  shore  prevents  a  Marine 
Expeditionary  Unit  from  executing  assault  missions  with 
tactical  surprise,  increases  the  time  to  close  combat  power 
ashore,  and  renders  the  unit  vulnerable  to  threats.  Testing 
showed  that  JLTVs  are  slower  to  load,  prepare  for  fording, 
and  transition  to  maneuver  ashore  than  HMMWV  due  to 
their  larger  vehicle  size,  delays  that  occur  while  awaiting 
suspension  mode,  and  other  vehicle  adjustments  (e.g.,  tire 
pressure). 

•  During  DT/OT,  the  JLTVs  demonstrated  more 
maneuverability  in  soft  soil  and  better  fording  capability 
than  HMMWV.  Crews  have  less  visibility  in  JLTVs  than 
HMMWV  because  of  smaller  windows,  placement  of  mission 
equipment,  and  positioning  of  window  panels. 

•  Analysis  of  the  JLTV  EMD  ballistic  test  data  is  ongoing. 
These  data  will  be  used  to  make  a  final  assessment  of 
threshold-level  force  protection  Key  Performance  Parameter 
for  all  three  vendors. 

Recommendations 

•  Status  of  Previous  Recommendations.  There  were  no 
previous  recommendations  for  this  program. 

•  FY 14  Recommendation. 

1 .  The  program  should  develop  a  plan  to  correct  JLTV 
performance  deficiencies  discovered  during  the  JLTV 
DT/OT  and  LUT  before  production. 
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Joint  Tactical  Network  (JTN) 


Executive  Summary 

•  In  January  2014,  the  Defense  Acquisition  Executive 
approved  the  Joint  Tactical  Networking  Center  (JTNC) 
charter  that  defines  the  responsibilities  of  the  Program 
Manager-Joint  Tactical  Networks  (PM- JTN)  to  develop  and 
maintain  the  Joint  Enterprise  Network  Manager  (JENM); 
report  on  program  acquisition  and  sustainment  status  to  the 
JTNC  Board  of  Directors;  and  develop  a  plan  to  transition 
waveform  development  to  the  Services  by  4QFY15. 

•  In  May  2014,  DOT&E  assessed  the  JENM  as  a  part  of  the 
Manpack  radio  FOT&E  during  the  Network  Integration 
Evaluation  (NIE)  14.2. 

-  Soldiers  using  JENM  were  able  to  plan  mission 
configuration  files  and  load  them  onto  Manpack  radios. 

-  The  network  planning  process  is  cumbersome  and 
loading  radios  is  too  slow  for  the  unit’s  operational 
tempo. 

-  Soldiers  were  able  to  monitor  the  network  using  JENM, 
but  the  monitoring  function  provides  little  operational 
utility  to  the  unit. 

•  PM-JTN  is  currently  developing  a  Test  and  Evaluation 
Master  Plan  (TEMP)  that  describes  testing  of  the  JENM  and 
waveforms  in  coordination  with  the  host  radio  programs. 

System 

•  The  PM-JTN  provides  software  applications  that  enable 
software-defined  radio  sets  to  provide  communications 
to  tactical  forces.  The  software  applications  include 
software-defined  waveforms  and  enterprise  network 
management. 

•  Software-defined  waveforms  are  loaded  into  and  considered 
a  part  of  a  radio  set.  JTN  is  responsible  for  the  following 
software-defined  waveforms: 

-  Soldier  Radio  Waveform  (SRW) 

-  Wideband  Networking  Waveform  (WNW) 

-  Single-Channel  Ground  and  Airborne  Radio  System 
(SINCGARS) 

-  Mobile  User  Objective  System  (MUOS) 

-  Link  16 

•  The  JENM  is  a  set  of  software  applications  that  allow  the 
Services  to  plan,  configure,  and  monitor  software-defined 
radio  networks.  The  JENM  is  separate  from  the  radio  sets 
and  is  deployed  on  a  Joint  Tactical  Networking  Environment 
Network  Operations  Toolkit  computer.  The  current 
JENM,  version  1.2.8. 1,  configures  SRW  radio  networks 
using  Manpack  and  Rifleman  radios.  Future  releases  will 
configure  radios  for  WNW  and  MUOS  networks. 


•  The  JENM  functions  include  planning,  loading,  monitoring, 

controlling,  and  reporting. 

-  The  planning  function  develops  network  parameters  and 
creates  a  Radio  Mission  Data  Set  (RMDS)  file. 

-  The  loading  function  transfers  the  RMDS  file  into  the 
radio  sets  to  configure  them  for  network  operation. 

-  The  monitoring  function  provides  a  near  real-time  display 
of  the  WNW  or  SRW  network  status  and  condition  of  the 
network  radios. 

-  The  controlling  function  allows  the  signal  Soldier  to  make 
changes  to  the  network,  to  include  sending  commands  to 
radio  operators,  changing  the  configuration  parameters 

of  the  radio  sets,  or  conducting  cryptographic  functions 
(rekey,  zeroize,  and  transfer). 

-  The  reporting  function  records  all  network  management 
events  and  logs  the  data  for  analysis. 
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Mission 

•  Forward-deployed  military  forces  use  the  software-defined 
radios  to  communicate  and  create  networks  to  exchange  voice, 
video,  and  data  during  all  aspects  of  tactical  military  operations. 

•  Signal  staffs  intend  to  use  the  JENM  to: 

-  Plan,  load,  monitor,  control,  and  report  on  network 
operations  involving  software-defined  radio  sets  running 
SRW  and  WNW. 


-  Load  the  RMDS  file  for  the  initial  provisioning  of  a  MUOS 
terminal  to  connect  to  a  MUOS  satellite  network. 

Major  Contractor 

The  Boeing  Company,  Phantom  Works  Division  -  Huntington 
Beach,  California 


Activity 

•  In  January  2014,  the  Defense  Acquisition  Executive  approved 
the  JTNC  charter,  which  defines  the  responsibilities  of  both 
the  JTNC  and  PM-JTN  following  the  dissolution  of  the 
Joint  Tactical  Radio  System  Network  Enterprise  Domain. 
PM-JTN  was  given  responsibility  to: 

-  Develop  and  maintain  the  JENM 

-  Report  on  program  acquisition  and  sustainment  status  to 
the  JTNC  Board  of  Directors 

-  Transition  waveform  development  and  sustainment 
responsibilities  to  the  designated  Services  (by  waveform) 
by  4QFY15 

•  From  January  through  February  2014,  the  Army  assessed 
JENM  loading  on  the  Manpack  radio  during  the  Manpack 
radio  Government  Development  Test  4  at  Fort  Huachuca, 
Arizona. 

•  During  the  May  2014  NIE  14.2,  the  Manpack  radio  FOT&E 
assessed  Soldiers  using  the  JENM  to  plan,  configure,  load,  and 
monitor  Manpack  radio  networks.  The  Army  conducted  this 
test  according  to  a  DOT&E -approved  test  plan. 

•  The  program  is  developing  a  TEMP  to  describe  testing 
activities  to  fulfill  the  responsibilities  described  within  the 
JTNC  charter. 

Assessment 

•  During  Government  Development  Test  4,  the  JENM: 

-  Allowed  Soldiers  to  load  and  reconfigure  radios  with 
four  of  five  threshold  waveforms  and  network  presets 

-  Demonstrated  partial  capability  to  monitor  a  radio  network 

-  Enabled  Soldiers  to  access  and  view  the  status  of  Manpack 
radio  configurations  and  Communications  Security 
(COMSEC)  keys 

•  During  the  Manpack  radio  FOT&E  at  NIE  14.2: 

-  Soldiers  using  JENM  were  able  to  plan  mission 
configuration  files  and  load  them  onto  Manpack  radios. 
JENM’s  planning  of  the  mission  configuration  files  was 
not  easy  and  required  several  attempts  before  success. 


Network  planners  encountered  difficulty  with  network 
complexity  due  to  the  large  number  and  multiple  variants 
of  networking  radios  within  Army  units. 

-  JENM  was  slow  when  loading  radios.  Soldiers  using 
JENM  and  a  Simple  Key  Loader  to  load  a  Manpack  radio 
with  mission  configuration  files  and  COMSEC  keys 
experienced  load  times  of  30  minutes  and  greater  per 
radio.  Loading  all  radios  within  a  company  can  take  as 
long  as  24  hours.  This  time  requirement  is  not  practical 
for  a  reload  that  must  occur  once  a  month,  per  COMSEC 
guidelines. 

-  JENM’s  network  monitoring  function  provided  little 
operational  utility  for  the  unit.  Companies  do  not  have  a 
Soldier  assigned  to  monitor  the  network  and  the  distance 
from  the  headquarters  to  the  radio  network  often  exceeded 
radio  transmission  range. 

-  One  signal  Soldier  per  company  was  not  sufficient 
to  accomplish  network  management,  operation,  and 
maintenance  required  for  the  unit’s  communications 
equipment.  Units  tasked  Soldiers  from  other  mission  areas 
to  assist. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 

report  for  this  program. 

•  FY 14  Recommendations.  The  Army  should: 

1 .  Complete  a  JTN  TEMP  that  describes  testing  of  the 
JENM  and  waveforms  (until  the  waveforms  transition  to 
designated  Services). 

2.  Correct  the  deficiencies  noted  during  the  Manpack  radio 
FOT&E  and  validate  corrections  during  future  testing. 

3.  Evaluate  the  force  structure  requirements  of  adding 
software-defined,  networking  radios  and  network 
management  responsibilities  into  company-level 
organizations. 
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M80A1  7.62  mm  Cartridge 


Executive  Summary 

•  Forces  will  use  the  M80A1  cartridge,  fired  by  several  different 
weapons  systems,  to  defeat  targets  with  improved  lethality 
compared  to  the  current  M80  ball  cartridge. 

•  The  Army  successfully  completed  live  fire  testing  of  the 
M80A1  in  July  2014.  DOT&E  assessed  the  M80A1  as  lethal. 

•  The  Army  authorized  fielding  in  September  2014. 

System 

•  The  7.62  mm  M80A1  cartridge  is  intended  to  replace  the 
lead- based  projectile  contained  in  the  current  M80  ball 
cartridge  with  a  projectile  utilizing  environmentally-friendly 
materials.  It  is  designed  to  provide  improved  lethality 
compared  to  the  current  M80  ball  cartridge. 

•  The  M80A1  cartridge  is  compatible  with  the  M240  series  of 
machine  guns,  the  M134  “mini  gun,”  the  Mk48  machine  gun, 
and  the  M110,  MK17,  M40A5,  and  M14  series  rifles.  This 
new  cartridge  is  intended  to  be  a  direct  replacement  for  the 
currently  fielded  M80  cartridge. 

•  The  M80A1  is  a  three-part  projectile  consisting  of  a  steel 
penetrator,  a  copper  slug,  and  a  reverse-drawn  copper  jacket. 


1  -  Assembled  Round 
(Pentetrator,  Slug,  and 

Copper  Jacket) 

2  -  Copper  Slug 

3  -  Steel  Penetrator 

4  -  Copper  Jacket 

5  -  Complete  Cartridge 


Mission 

Forces  equipped  with  weapons  that  fire  the  M80A1  will  engage 
enemy  combatants  during  tactical  operations  in  accordance  with 
applicable  tactics,  techniques,  and  procedures  to  accomplish 
assigned  missions. 


Major  Contractors 

•  Alliant-Techsystems,  Small  Caliber 
Systems  -  Independence,  Missouri 

•  Olin  Winchester  -  East  Alton,  Illinois 


Activity 

•  The  Army  successfully  completed  live  fire  testing  of  the 
M80A1  in  July  2014.  Testing  was  conducted  in  accordance 
with  the  DOT&E-approved  live  fire  strategy  and  test  plans. 

•  The  Army  used  gelatin  targets  to  obtain  data  as  inputs  for 
complex  computer  modeling  of  M80A1  performance.  Testing 
also  included  shots  against  light  material  barriers  and  other 
targets  to  determine  the  projectile’s  ability  to  perforate  the 
target. 

•  The  Army  approved  fielding  of  the  M80A1  as  an  Enhanced 
Perfonnance  Round  via  an  Engineering  Change  Proposal  to 
the  M80  in  September  2014. 

•  DOT&E  will  publish  a  classified  lethality  report  for  the 
M80A1  in  January  2015. 


Assessment 

•  The  M80A1  demonstrated  adequate  performance  and  lethality. 

•  The  classified  DOT&E  lethality  report  provides  a  detailed 
description  of  the  M80Al’s  performance. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations.  None 
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M109  Family  of  Vehicles  (FoV) 
Paladin  Integrated  Management  (PIM) 


Executive  Summary 

•  In  FY14,  the  Army  conducted  multiple  test  phases  as  part 
of  the  M109  Family  of  Vehicles  (FoV)  Paladin  Integrated 
Management  (PIM)  LFT&E  program,  including  follow-on 
ballistic  hull  testing,  component  ballistic  tests,  and  fire 
survivability  testing. 

•  The  assessment  of  results  for  each  of  the  test  phases  is 
ongoing.  Preliminary  assessments  of  the  fire  survivability 
tests  indicate  potential  vulnerability  issues  with  one  PIM 
vehicle’s  automatic  fire  extinguishing  system. 

•  Underbody  blast  testing  will  not  be  accomplished  until 
high-fidelity  prototypes  and  full-up  systems  are  available. 

System 

•  The  M109  FoV  PIM  consists  of  two  vehicles:  the 
Self-Propelled  Howitzers  (SPH)  and  Carrier,  Ammunition, 
Tracked  (CAT)  resupply  vehicles. 

-  The  M109  FoV  SPH  is  a  tracked,  self-propelled  155  mm 
howitzer  designed  to  improve  sustainability  over  the 
legacy  M109A6  howitzer  fleet.  The  full-rate  production 
howitzers  will  have  a  newly  designed  hull,  modified 
Bradley  Fighting  Vehicle  power  train  and  suspension, 
and  a  high-voltage  electrical  system.  The  SPH  is 
operated  by  a  crew  of  four  and  can  engage  targets  at 
ranges  of  22  kilometers  using  standard  projectiles  and 
30  kilometers  using  rocket-assisted  projectiles. 

-  The  M109  FoV  CAT  supplies  the  SPH  with  ammunition. 
The  full-rate  production  ammunition  carriers  will  have  a 
common  chassis  with  the  SPH.  The  ammunition  carriers 
are  designed  to  carry  12,000  pounds  of  ammunition  in 
various  configurations  and  a  crew  of  four  Soldiers. 

•  The  Army  will  equip  the  SPH  and  CAT  with  two  armor 
configurations  to  meet  two  threshold  requirements  for 
force  protection  and  survivability  -  Threshold  1  (Tl)  and 
Threshold  2  (T2). 

-  The  base  T 1  armor  configuration  is  integral  to  the  SPH 
and  CAT.  The  T2  configuration  is  intended  to  meet 
protection  requirements  beyond  the  T 1  threshold  with 
add-on  armor  kits. 


-  The  Army  plans  to  employ  PIM  vehicles  in  the  Tl 

configuration  during  normal  operations  and  will  equip  the 
SPH  and  CAT  with  T2  add-on  armor  kits  during  combat 
operations. 

•  The  M109  FoV  SPH  can  fire  the  PGK  and  the  Excalibur 
precision  munition  to  increase  delivery  accuracy.  The  Army 
developed  the  PGK  to  reduce  the  dispersion  of  unguided 
projectiles  and  the  Excalibur  precision  munition  to  provide 
Field  Artillery  units  a  precision  engagement  capability. 

•  The  Army  intends  to  employ  the  M109  FoV  as  part  of  a  Fires 
Battalion  in  the  Armored  Brigade  Combat  Team  and  Artillery 
Fires  Brigades  with  the  capability  to  support  any  Brigade 
Combat  Team. 

•  The  Army  plans  to  field  up  to  557  sets  of  the  M109  FoV  with 
full-rate  production  planned  for  FY17. 

Mission 

Field  Artillery  units  employ  the  M109  FoV  to  destroy,  defeat,  or 

disrupt  the  enemy  by  providing  integrated,  massed,  and  precision 

indirect  fire  effects  in  support  of  maneuver  units  conducting 

unified  land  operations. 

Major  Contractor 

BAE  Systems  -  York,  Pennsylvania 


Activity 

•  The  Army  conducted  multiple  test  phases  as  part  of  the 
M109  FoV  PIM  LFT&E  program  at  Aberdeen  Proving 
Ground,  Maryland,  including  follow-on  ballistic  hull  testing, 
component  ballistic  tests,  and  fire  survivability  testing. 


In  FY13,  initial  exploitation  testing  against  SPH  and 
CAT  ballistic  structures  to  characterize  the  protection 
they  provide  against  specified  threats  revealed  multiple 
vulnerable  areas  in  the  systems’  Tl  and  T2  armor 


Ml 09  FoV  PIM 


121 


F  Y  1  4  ARMY  PROGRAMS 


configurations.  The  Program  Office  addressed  each  of 
these  vulnerabilities,  and  proposed  corrective  actions 
for  most  of  them.  In  3QFY 1 4,  the  Army  conducted 
follow-on  ballistic  hull  (retrofit)  testing  to  validate 
proposed  corrective  actions. 

-  In  3QFY14,  the  Army  conducted  component  ballistic 
tests  of  the  SPH  and  CAT  high-voltage  electrical  systems 
to  characterize  their  performance  and  unexpected 
vulnerabilities  after  being  subjected  to  a  threat  impact. 

-  In  4QFY14,  the  Army  conducted  fire  survivability  testing 
of  the  SPH.  The  Army  will  complete  all  fire  survivability 
testing  by  May  2015. 

•  The  Army  conducted  all  test  phases  in  accordance  with 
DOT&E-approved  test  plans. 

•  The  Army  has  designed  and  will  test  a  separate  underbelly 
kit  (not  a  component  of  the  T 1  and  T2  armor  configurations) 
to  determine  the  potential  protection  an  SPH  and  CAT  can 
provide  against  IEDs  similar  to  those  encountered  in  Iraq  and 
Afghanistan.  That  testing  is  expected  to  occur  in  FY15-16 
when  the  high-fidelity  prototypes  and  full-up  systems  are 
available. 


Assessment 

•  Assessment  of  results  from  the  follow-on  SPH  and  CAT  ballistic 
hull  testing  and  component  ballistic  tests  of  the  high  voltage 
electrical  systems  on  the  systems  is  ongoing.  DOT&E  will 
include  that  assessment  in  a  final  LFT&E  report  to  Congress. 

•  Preliminary  results  of  the  SPH’s  fire  survivability  tests  indicate 
potential  vulnerability  issues  with  the  platform’s  automatic 
fire  extinguishing  system.  The  specific  details  are  classified. 
DOT&E  will  include  that  assessment  in  a  final  LFT&E  report 
to  Congress. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  has 
successfully  addressed  one  previous  recommendation 
and  made  progress  on  two.  However,  they  still  need  to 
implement  and  validate  planned  armor  configuration  changes 
for  Low-Rate  Initial  Production  vehicles  prior  to  full-up 
system-level  testing. 

•  FY 14  Recommendation. 

1 .  The  Army  should  correct  the  deficiencies  identified  in  fire 
survivability  testing  and  validate  those  fixes  in  test. 
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M829E4  Armor  Piercing,  Fin  Stabilized, 
Discarding  Sabot  -  Tracer  (APFSDS-T) 


Executive  Summary 

•  The  M829E4  120  mm  cartridge  is  a  line-of-sight  kinetic 
energy  cartridge  designed  for  the  Abrams  main  battle  tank. 

•  During  Integrated  Test  and  Evaluation  (1T&E),  in-bore 
structural  failures  (IBSFs)  occurred  in  the  ambient  zone 
temperature  range  (60-86  degrees  Fahrenheit).  The  program 
manager  stopped  testing  and  conducted  a  formal  failure 
analysis. 

•  Preliminary  testing  indicates  the  M829E4  cartridge  is 
demonstrating  an  overall  reliability  of  94  percent  as  a  result  of 
IBSFs.  The  reliability  requirement  is  98  percent. 

•  The  failure  analysis  identified  reliability  problems  with 
the  production  process  of  the  cartridge.  The  program  has 
implemented  production  process  changes  to  address  reliability 
deficiencies  and  is  in  the  process  of  conducting  additional 
testing  to  verify  the  effectiveness  of  the  changes. 

•  On  June  30,  2014,  the  Program  Executive  Officer  Ammunition 
approved  the  M829E4  cartridge  for  Milestone  C  with 
provisions. 

•  The  Milestone  Decision  Authority  authorized  910  M829E4 
cartridges  for  low-rate  initial  production  (LRIP)  on 

June  30,  2014. 

System 

•  The  M829E4  120  mm  cartridge  is  a  line-of-sight  kinetic 
energy  cartridge  designed  for  the  Abrams  main  battle  tank.  It 
is  the  materiel  solution  for  the  Abrams’  lethality  capability 
gap  against  threat  vehicles  equipped  with  3rd  Generation 
Explosive  Reactive  Armor. 

•  The  M829E4  cartridge  is  an  Armor-Piercing,  Fin-Stabilized, 
Discarding  Sabot,  with  Tracer  cartridge  consisting  of  a 
depleted  uranium  long-rod  penetrator  with  a  three-petal 
composite  sabot. 

•  The  flight  projectile  includes  a  low-drag  fin  with  a  tracer, 
windshield,  and  tip  assembly. 

•  The  propulsion  system  of  the  M829E4  cartridge  is  a 
combustible  cartridge  case  similar  to  that  of  the  currently 
fielded  suite  of  Abrams’  120  mm  tank  cartridges. 

•  The  M829E4  has  comparable  characteristics  to  its  predecessor, 
the  M829A3,  in  length,  weight,  and  center  of  gravity. 


Ammunition  Data  Link 
(ADL)  interface  rings 
on  base  of  the 
M829E4 


The  visible  difference  between  the  two  cartridges  is  the 
Ammunition  Data  Link  (ADL)  interface  rings  on  the  base 
of  the  M829E4.  The  rings  serve  as  the  interface  between 
the  Abrams’  fire  control  system  and  the  M829E4.  The  ADL 
enables  the  Abrams’  fire  control  system  to  send  information  to 
the  M829E4. 

Mission 

Armored  Brigade  Combat  Teams  equipped  with  the  M829E4 
120  mm  cartridge  will  have  the  ability  to  defeat  current  and 
projected  threat  main  battle  tanks  equipped  with  third  generation 
explosive  reactive  armor  and  active  protection  systems.  The 
Army  intends  the  M829E4  to  provide  enhanced  lethality  beyond 
its  predecessor,  the  M829A3,  and  will  enhance  the  Joint  Forces 
Commander’s  capability  to  conduct  decisive  operations  during 
Unified  Land  Operations. 

Major  Contractor 

Alliant  Techsystems  Inc.  (ATK)  -  Plymouth,  Minnesota 


Activity 

•  ATEC  conducted  a  Design  Engineering  Test  consisting  of 
50  M829E4  cartridges  at  Yuma  Test  Center,  Arizona,  from 
June  through  November  2013. 


ATEC  conducted  two  phases  of  1T&E  at  Yuma  Test  Center; 
Redstone  Test  Center,  Alabama;  and  Aberdeen  Test  Center, 
Maryland,  from  December  2013  through  mid-June  2014. 
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-  During  Phase  1,  IBSFs  occurred.  The  program 
manager  stopped  testing  and  conducted  a  formal  failure 
investigation. 

-  The  Army  produced  subsequent  M829E4  cartridges  that 
incorporated  design  configuration  and  production  process 
changes  to  correct  the  IBSFs. 

-  During  Phase  2  of  1T&E,  100  M829E4  cartridges  with  the 
design  configuration  and  production  process  changes  were 
fired.  IBSFs  continued  to  occur.  The  program  manager 
again  stopped  testing  and  conducted  another  formal  failure 
investigation. 

•  ATEC  conducted  vulnerability  live  fire  testing  on  the  Abrams 
uploaded  with  M829E4  cartridges  to  ensure  that  stored 
cartridges  do  not  pose  a  threat  when  an  Abrams  tank  is 
engaged  by  enemy  fire  from  March  through  June  2014. 

•  ATEC  conducted  DOT&E-approved  live  fire  tests  to  support  a 
lethality  assessment  from  June  16  -  25,  2014. 

-  Due  to  the  IBSFs  observed  during  Phase  2  of  1T&E,  the 
Army  ceased  live  fire  testing.  The  potential  for  an  IBSF 
during  a  live  fire  test  presented  an  unacceptable  risk  to  the 
live  fire  facility. 

-  The  Army  will  resume  live  fire  testing  when  the  risk  to 
the  live  fire  test  facility  related  to  IBSFs  is  mitigated  and 
production-representative  rounds  are  available. 

•  The  Army  conducted  the  M829E4  User  Excursion  at 
Aberdeen  Proving  Ground,  Maryland,  in  accordance  with 
a  DOT&E-approved  Test  and  Evaluation  Master  Plan  on 
May  22,  2014. 

-  The  User  Excursion  utilized  an  ADL-equipped  Abrams 
tank  and  qualified  Abrams  tank  crew,  who  performed  select 
engagements  of  a  Tank  Table  VI  gunnery  event. 

-  The  gunnery  event  involved  eight  M829E4  engagements 
against  moving  and  stationary  targets. 

-  The  User  Excursion  was  designed  to  confirm  M829E4 
cartridge  transparency  of  employment  and  provide 
supplemental  data  for  accuracy,  performance,  and 
reliability. 

•  The  Program  Executive  Officer  Ammunition  approved 
the  M829E4  cartridge  for  Milestone  C  with  provisions  on 
June  30,  2014. 

-  The  provisions  require  that  the  Army  investigate  fixes 
to  the  failures  identified  in  IT&E,  and  verify  those 
fixes  in  ATEC-approved  follow-on  testing  prior  to  the 
Full-Rate  Production  Decision  Review/Material  Release, 
per  the  Milestone  C  Acquisition  Decision  Memorandum. 

•  The  Milestone  Decision  Authority  authorized  910  M829E4 
cartridges  for  LRIP  on  June  30,  2014. 

-  The  Acquisition  Decision  Memorandum  directed  additional 
reliability  testing  to  validate  that  LRIP  cartridges  meet 
reliability  requirements. 

•  The  Army  conducted  another  formal  failure  investigation 
into  the  failures  experienced  during  the  second  phase  of 
1T&E  and  identified  potential  production  process  and  design 


configuration  changes  to  correct  the  IBSFs  from  June  through 
August  2014. 

-  Results  of  the  formal  failure  investigation  led  the  Army 
to  produce  two  configurations  (Configuration  A  and 
Configuration  B)  M829E4  cartridges  to  correct  IBSFs. 

-  Configuration  A  incorporated  four  production  process 
changes. 

-  Configuration  B  incorporated  two  design  configuration 
changes  and  the  four  Configuration  A  production  process 
changes. 

•  The  Army  conducted  testing  (Verification  #1)  at  Yuma  Proving 
Ground,  Arizona,  involving  Configuration  A  and  B  cartridges 
on  October  27,  2014. 

-  Thirty  events  employing  Configuration  A  cartridges.  All 
30  Configuration  A  cartridges  fired  successfully. 

-  Twenty-six  events  employing  Configuration  B  cartridges. 
Two  of  the  26  Configuration  B  cartridges  experienced 
IBSF,  and  24  cartridges  fired  successfully. 

•  Based  on  Verification  #1  test  results,  the  Army  approved 
the  Configuration  A  cartridge  for  Verification  #2  testing  in 
February  2015. 

-  After  Verification  #2  testing  is  complete  and  yields 
successful  results,  the  Army  will  resume  live  fire  testing, 
production,  and  First  Article  Acceptance  Testing. 

Assessment 

•  Preliminary  testing  indicates  the  M829E4  cartridge  is 
demonstrating  an  overall  reliability  of  94  percent.  The 
reliability  requirement  is  98  percent. 

-  During  IT&E  testing,  the  cartridge  experienced  IBSFs 
in  the  ambient  zone  temperature  range.  The  M829E4 
cartridge  reliability  was  assessed  by  firing  the  cartridge 
within  three  temperature  zone  conditions:  cold  (-25  to 
19  degrees  Fahrenheit),  ambient  (60  to  86  degrees 
Fahrenheit),  and  hot  (120  to  145  degrees  Fahrenheit). 

-  In  ambient  zone  temperatures,  the  cartridge  demonstrated 
91.6  percent  reliability.  In  cold  zone  temperatures,  the 
cartridge  demonstrated  94.6  percent  reliability,  and  in  hot 
zone  temperatures,  the  cartridge  demonstrated  97.2  percent 
reliability.  Overall  reliability  is  94  percent. 

-  The  program  has  proposed  additional  design  configuration 
changes  to  address  reliability  deficiencies  and  is  in  the 
process  of  testing  to  verify  the  effectiveness  of  the  changes. 

•  Effectiveness  is  based  on  the  Single  Shot  Probability  of  Kill 
metric  computed  using  the  Passive  Vehicle  Target  Model 
(PVTM).  Cartridge  reliability  is  an  input  to  PVTM,  along 
with  accuracy  and  lethality.  Because  reliability  is  an  input 
to  the  PVTM  model,  effectiveness  is  sensitive  to  reliability 
results. 

•  DOT&E  will  submit  an  evaluation  after  completion  of  the 
production  process  changes  and  additional  testing. 
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Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations.  The  Army  should: 

1.  Conduct  an  FOT&E  event  for  the  M829E4  and  Abrams’ 
production-representative  hardware  and  software  as  part 
of  the  FY16  Ammunition  Data  Link  Qualification  event  at 
Aberdeen  Proving  Ground,  Maryland. 

2.  Conduct  Verification  #2  testing  to  demonstrate  that 
production  process  changes  enable  the  cartridge  to  meet 
reliability  requirements  in  a  DOT&E-approved  test  prior  to 
the  Full  Material  Release  decision. 


3.  Complete  live  fire  tests  specified  in  the  live  fire  strategy. 

If  major  design  changes  occur  to  the  M829E4  cartridge  to 
correct  IBSFs  experienced  during  IT&E,  the  Army  should 
conduct  live  fire  tests  germane  to  the  cartridge’s  design 
changes. 

4.  Update  PVTM  with  results  from  Verification  #2  testing 
and  the  remaining  live  fire  tests,  and  use  the  updated 
effectiveness  model  to  evaluate  M829E4  lethality  against 
relevant  threat  targets. 
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Manpack  Radio 


Executive  Summary 

•  In  December  2013,  the  Defense  Acquisition  Executive  (DAE) 
approved  an  additional  low-rate  initial  production  (LR1P)  lot 
of  1,500  Manpack  radios.  This  brought  the  total  number  of 
Manpack  radios  procured  through  LRIP  to  5,326. 

•  In  April  through  May  2014,  the  Army  conducted  a 
Manpack  radio  FOT&E  as  part  of  the  Network  Integration 
Evaluation  14.2  at  Fort  Bliss,  Texas,  in  accordance  with  a 
DOT&E-approved  test  plan.  DOT&E  assessed  the  Manpack 
radio  as  not  operationally  effective  when  employed  in 
dismounted  operations,  operationally  effective  for  mounted 
operations,  and  not  operationally  suitable. 

•  In  May  2014,  the  DAE  approved  a  new  Manpack  radio 
acquisition  strategy.  This  strategy  specifies  that  the  Army 
will  conduct  a  full  and  open  competition,  open  to  all  industry 
vendors,  for  the  full-rate  production  phase  of  the  Manpack 
radio  program.  The  Army  is  currently  developing  a  Test  and 
Evaluation  Master  Plan  (TEMP)  to  support  the  new  Manpack 
radio  acquisition  strategy. 

System 

•  The  Army’s  Handheld,  Manpack,  and  Small  Form  Fit  (HMS) 
program  evolved  from  the  Joint  Tactical  Radio  System 
program  and  provides  software-programmable  digital  radios 
to  support  the  Army’s  tactical  communications  requirements. 
The  two  radios  that  comprise  the  HMS  program  are  the 
Manpack  radio  and  the  Rifleman  Radio.  (Rifleman  Radio  is  a 
separate  program  of  record,  with  a  separate  article  later  in  this 
Annual  Report.) 

•  The  Manpack  radio  is  a  two-channel  software-defined  radio 
capable  of  transmitting  both  voice  and  data  communications. 

•  Manpack  radio  has  an  operating  spectrum  of  2  to 

512  Megahertz  with  a  20-watt  power  output,  or  512  Megahertz 
to  2.5  Gigahertz  with  a  10-watt  power  output. 

•  Manpack  radio  is  capable  of  employing  multiple  waveforms: 

-  Soldier  Radio  Waveform  (SRW) 


-  Single  Channel  Ground  and  Airborne  Radio  System 
(SINCGARS)  waveform 

-  Ultra-High  Frequency  Satellite  Communications 
(UHF-SATCOM)  18  IB 

-  Demand  Assigned  Multi- Access  UHF -SAT COM 
waveforms  182Aand  183A 

•  The  Services  intend  the  Manpack  radio  to  eventually  host  the 
Mobile  User  Objective  Satellite  waveform,  which  is  still  in 
development. 

Mission 

Army  tactical  units  will  use  Manpack  radios  to  provide 
networked  voice  and  data  communications  during  all  tactical 
operations.  The  Manpack  radio  can  operate  in  one  of  three 
modes:  dismounted,  vehicle  mounted,  or  mounted  in  a  kit  for  use 
in  a  Tactical  Operations  Center. 

Major  Contractors 

•  General  Dynamics,  C4  Systems  -  Scottsdale,  Arizona 

•  Rockwell  Collins  -  Cedar  Rapids,  Iowa 


Activity 

•  In  December  2013,  the  DAE  approved  an  additional  LRIP  lot 
of  1,500  Manpack  radios.  This  brought  the  total  number  of 
Manpack  radios  procured  through  LRIP  to  5,326. 

•  In  January  through  February  2014,  the  Army  conducted 
Government  Developmental  Test  4  (GDT  4)  at  Fort  Huachuca, 
Arizona. 

-  GDT  4  tested  corrective  actions  applied  to  the 
Manpack  radio  in  response  to  deficiencies  identified 
in  previous  test  events,  including  the  Multi-Service 
Operational  Test  and  Evaluation  conducted  in  May  2012. 


Among  the  previously  identified  deficiencies  were  both 
unsatisfactory  SINCGARS  waveform  performance  and 
poor  reliability. 

-  The  test  was  also  designed  to  assess  capabilities  that  had 
not  been  previously  tested.  These  additional  capabilities 
included  route  and  retransmission  and  additional 
satellite  communications  waveforms  (Demand  Assigned 
Multi-Access  UHF-SATCOM  182Aand  183A).  Route 
and  retransmission  refers  to  the  radio  function  that 
enables  traffic  coming  into  the  radio  on  one  channel  to  be 
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“cross-banded”  within  the  radio  and  re-transmitted  out  on 
the  other  channel,  either  on  the  same  waveform  or  on  a 
different  waveform. 

•  In  April  through  May  2014,  the  Army  conducted  a 
Manpack  radio  FOT&E  as  part  of  the  Network  Integration 
Evaluation  14.2  at  Fort  Bliss,  Texas,  in  accordance  with  a 
DOT&E-approved  test  plan.  During  this  FOT&E,  infantry 
and  cavalry  units  employed  the  Manpack  radio  in  executing 
both  mounted  and  dismounted  tactical  operations.  The 
purpose  of  this  test  was  to  evaluate  in  a  realistic  operational 
environment  the  correction  of  Manpack  radio  deficiencies 
identified  in  the  Multi-Service  Operational  Test  and 
Evaluation,  as  well  as  the  additional  capabilities  that  had  yet 
to  be  demonstrated  in  an  operational  test. 

•  In  May  2014,  the  DAE  approved  a  new  Manpack  radio 
acquisition  strategy.  This  strategy  specifies  that  the  Army 
will  conduct  a  full  and  open  competition,  open  to  all 
industry  vendors,  for  the  full-rate  production  phase  of  the 
Manpack  radio  program. 

•  The  Army  is  currently  developing  a  TEMP  to  support  the  new 
Manpack  acquisition  strategy. 

Assessment 

•  Based  upon  the  results  of  the  2014  FOT&E,  DOT&E  made 
the  following  assessment: 

-  The  Manpack  radio  is  not  operationally  effective  when 
employed  in  dismounted  operations.  This  assessment 
is  the  result  of  the  SRW  providing  insufficient  range 

to  support  dismounted  company  and  platoon-level 
operations.  SRW  is  the  waveform  the  Army  has  selected 
for  both  voice  and  data  communications  at  these  tactical 
echelons.  SRW  demonstrates  a  shorter  range  vis-a-vis 
the  legacy  SINCGARS  waveform  it  is  replacing  in 
this  network.  This  result  is  not  surprising  given  that 
SRW  operates  at  a  much  higher  frequency  than  does 
SINCGARS.  Higher  frequencies  have  shorter  ranges  and 
are  more  affected  by  terrain  obstructions. 

-  The  Manpack  radio  is  operationally  effective  for  mounted 
operations.  As  tested,  vehicular-mounted  Manpack  radios 
overcame  the  SRW  range  limitations  due  to  higher  vehicle 
antennas.  Additionally,  the  mounted  test  unit  had  a  higher 
density  of  radios,  and  hence  a  denser  network,  aiding 
communications  in  comparison  to  the  dismounted  test 
unit. 

-  The  Manpack  radio  is  not  operationally  suitable.  The 
radio’s  weight  hinders  dismounted  operations.  For  a 


24-hour  operation,  the  Manpack  radio  weighs — inclusive 
of  batteries  and  antennas — approximately  three  times  what 
the  legacy  SINCGARS  Advance  System  Improvement 
Program  radio  weighs  (35  pounds  versus  12  pounds).  The 
Manpack  radio  generates  heat  at  a  level,  which,  while 
technically  meeting  the  Military  Standard  for  prolonged 
exposure,  adversely  affected  dismounted  operators. 

During  the  FOT&E,  SRW  on  the  Manpack  radio  was 
reliable,  exceeding  its  reliability  requirement  of  477-hour 
Mean  Time  Betwen  Essential  Function  Failure.  Although 
the  SINCGARS  waveform  did  not  meet  its  reliability 
requirement,  its  reliability  did  not  adversely  affect  the  test 
unit’s  ability  to  execute  the  mission.  SATCOM  waveform 
183  A  also  did  not  meet  its  reliability  requirement. 
SATCOM  waveforms  181B  and  182Ahad  insufficient 
operating  time  during  the  FOT&E  to  reach  a  conclusion  as 
to  their  reliability. 

•  During  GDT  4: 

-  All  Manpack  waveforms  performed  satisfactorily  with 
respect  to  voice  quality  and  data  speed  of  service. 

-  The  Manpack  radio  successfully  demonstrated 
simultaneous  dual-channel  operations,  with  the  exception 
of  the  SINCGARS-SINCGARS  waveform  combination, 
which  demonstrated  an  unsatisfactory  call  completion  rate 
and  poor  voice  quality. 

-  The  Manpack  radio  successfully  demonstrated  its  route  and 
retransmission  capability  for  both  voice  and  data,  with  the 
exception  of  SINCGARS  voice  route  and  retransmission. 
SINCGARS  voice  was  not  tested  due  to  radio  interference 
emanating  from  unrelated  testing  nearby. 

-  Manpack  radio  reliability  was  below  the  requirement, 
demonstrating  a  Mean  Time  Between  Essential  Function 
Failure  of  217  hours  versus  the  requirement  of  477  hours. 

-  Test  scenarios  were  considerably  less  challenging  than 
those  experienced  during  the  FOT&E.  For  example,  radio 
operations  were  largely  conducted  from  static  positions 
with  good  line-of-sight. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed 

the  FY13  recommendations. 

•  FY 14  Recommendation. 

1 .  The  Army  should  develop  a  TEMP  that  satisfactorily 
addresses  the  developmental  and  operational  testing 
supporting  the  new  Manpack  radio  acquisition  strategy. 
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Mine  Resistant  Ambush  Protected  (MRAP) 
Family  of  Vehicles  (FoV)  -  Army 


Executive  Summary 

•  The  Army  will  retain  8,585  Mine  Resistant  Ambush  Protected 
(MRAP)  vehicles  in  its  enduring  fleet  that  includes  MaxxPro 
Dash,  MaxxPro  Ambulance,  and  MRAP  All-Terrain  Vehicle 
(M-ATV)  variants. 

•  Early  live  fire  testing  conducted  in  4QFY14  suggests  that 
the  MaxxPro  Ambulance  variant  will  meet  its  required  level 
of  performance;  however,  additional  testing  remains  to  be 
executed  in  FY15. 

•  The  MaxxPro  Dash  with  Independent  Suspension  System 
(ISS)  and  the  MaxxPro  Survivability  Upgrade  (MSU)  kit 
installed  provides  protection  beyond  the  MRAP  Capabilities 
Production  Document  1.1  requirements. 

System 

•  The  MRAP  Family  of  Vehicles  (FoV)  is  designed  to  provide 
increased  crew  protection  and  vehicle  survivability  against 
current  battlefield  threats,  such  as  IEDs,  mines,  small  arms 
fire,  rocket-propelled  grenades,  and  explosively  formed 
penetrators.  MRAPs  are  employed  by  units  in  current 
combat  operations  in  the  execution  of  missions  previously 
accomplished  with  the  High  Mobility  Multi-purpose  Wheeled 
Vehicle. 

•  In  FY14,  the  MRAP  Joint  Program  Office  was  dissolved, 
and  the  Army  and  the  Marine  Corps  became  the  lead 
Services  responsible  for  their  respective  MRAP  variants. 

In  2013,  the  Army  defined  its  enduring  MRAP  fleet  that  it 
will  retain  post-transition  from  a  Joint  Program  Office  to  an 
Army-led  program  manager.  Per  the  Army  MRAP  III  study, 
8,585  MRAP  vehicles  will  be  retained  in  the  Army  enduring 
fleet: 

-  MaxxPro  Dash  with  ISS  and  MSU  kit  (2,633  vehicles), 

-  MaxxPro  Long  Wheel  Base  (LWB)  Ambulance  with  ISS 
(301  vehicles) 

-  M-ATV  with  Underbelly  Improvement  Kit 
(5,651  vehicles). 

•  This  report  covers  the  MaxxPro  Dash  with  MSU  and  the 
MaxxPro  LWB  Ambulance. 

-  The  Dash  variant  with  the  MSU  kit  is  a  shortened  version 
of  the  MaxxPro  LWB  variant.  The  Dash  variant  is 
designed  to  provide  improved  underbody  blast  protection, 
can  transport  six  persons,  and  is  equipped  with  an  ISS. 

-  The  MaxxPro  LWB  Ambulance  variant  is  designed  to 
transport  a  driver  and  two  crewmembers  (one  of  which 
is  a  medic)  with  the  ability  to  carry  a  combination  of 
two  litter-bound  or  four  ambulatory  patients.  This  variant 
is  equipped  with  an  ISS. 


MaxxPro  Long  Wheel 
Base  (LWB)  Ambulance 


MaxxPro  Dash  with 
MaxxPro  Survivability 
Upgrade  (MSU) 


Mission 

Multi-service  and  special  operations  units  equipped  with  the 
MRAP  FoV  conduct  mounted  patrols,  convoy  patrols,  convoy 
protection,  reconnaissance,  and  communications,  as  well  as 
command  and  control  missions  to  support  combat  and  stability 
operations  in  highly-restricted  rural,  mountainous,  and  urban 
terrain. 

•  Units  equipped  with  Dash  vehicles  will  conduct  small 
unit  combat  operations  such  as  mounted  patrols  and 
reconnaissance. 
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•  Units  equipped  with  the  MaxxPro  LWB  Ambulance  variant  Major  Contractor 

will  provide  enhanced  medical  evacuation  capabilities  with  Navistar  Defense  -  Warrenville,  Illinois 
protection  against  ballistic  threats  that  may  be  encountered 
while  transporting  Soldiers  during  evacuation. 


Activity 

MaxxPro  Dash  with  MSU 

•  The  Army  completed  the  LFT&E  of  the  MSU-equipped 
Dash  in  3QFY14.  This  testing  focused  on  the  underbody 
blast  threat.  The  results  from  the  legacy  Dash  LFT&E 
test  program  relative  to  other  tested  threats  such  as 
IEDs,  indirect  fire,  and  small  arms,  are  applicable  to  the 
MSU-equipped  Dash. 

MaxxPro  LWB  Ambulance 

•  The  Army  commenced  LFT&E  on  the  MaxxPro  LWB 
Ambulance  in  4QFY14  and  completed  in  1QFY15. 
LFT&E  of  the  MaxxPro  LWB  Ambulance  is  focused 
on  the  changes  to  the  vehicle  specific  to  the  ambulance 
configuration,  including  the  patient  litter  and  medical 
equipment. 

•  The  Army  conducted  all  testing  in  accordance  with 
DOT&E-approved  test  plans. 

Assessment 

MaxxPro  Dash  with  MSU 

•  Early  LFT&E  of  the  MSU  equipped  Dash  revealed 
problems  with  MSU  kit  integration  that  required 
engineering  changes  to  the  platform.  Testing  and 


evaluation  of  changes  to  the  MSU  kit  to  address  these 
problems  is  complete,  and  the  solution  will  be  integrated 
into  affected  vehicles  during  their  reset. 

•  The  MSU  kit  performs  as  intended,  mitigating  certain 
structural  issues  identified  from  combat  events. 

•  The  MSU-equipped  Dash  provides  underbody  blast 
protection  well  beyond  the  original  MRAP  Capabilities 
Production  Document  1.1  requirements. 

MaxxPro  LWB  Ambulance 

•  Testing  is  ongoing,  but  early  LFT&E  indicates  the  vehicle 
provides  protection  at  its  required  levels.  Additional  testing 
and  analysis  are  required  before  a  full  evaluation  can  be 
made. 

Recommendations 

•  Status  of  Previous  Recommendations.  There  was  no  live 
fire  or  operational  testing  conducted  on  the  M-ATV  in  FY14; 
therefore,  none  of  the  FY13  recommendations  apply  to  the 
vehicles  tested  in  FY14. 

•  FY 14  Recommendations.  None. 
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Nett  Warrior 


Executive  Summary 

•  In  3QF  Y 14,  the  Assistant  Secretary  of  the  Army  for 
Acquisition,  Logistics,  and  Technology  authorized 
an  additional  low-rate  initial  production  (LRIP)  for 

5,115  Nett  Warrior  systems  to  support  fielding  to  Capability 
Set  15  and  Capability  Set  16. 

•  The  Army  Test  and  Evaluation  Command  (ATEC)  executed  an 
IOT&E  in  two  phases: 

-  A  mounted  Calvary  troop  executed  Phase  1  during 
Network  Integration  Evaluation  14.2  at  Fort  Bliss,  Texas 
(May  2014). 

-  A  dismounted  rifle  company  executed  Phase  2  at 
Fort  Polk,  Louisiana  (November  2014). 

•  Nett  Warrior  improved  situational  awareness  at  the  platoon 
level  and  continues  to  enhance  pre-mission  planning  tasks, 
land  navigation,  and  command  and  control. 

•  Nett  Warrior  is  reliable,  demonstrating  a  reliability  point 
estimate  of  445  operating  hours  during  Phase  1  and  a 
reliability  point  estimate  of  175.4  operating  hours  during 
Phase  2. 

•  DOT&E  intends  to  publish  an  IOT&E  report  in  2QFY15  to 
support  the  3QFY15  Full-Rate  Production  Decision  Review. 


Support  Equipment 
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messages,  and  enemy  activity  on  a  digital  geo-referenced 
map  image.  The  Nett  Warrior  is  connected  through  a  secure 
radio  to  the  Soldier  Radio  Waveform  (SRW)  network  to 
communicate  among  different  echelons  using  voice,  data,  and 
position  location  information  messages. 


System 

•  The  Nett  Warrior  is  a  dismounted  leader  situational  awareness 
system  for  use  during  combat  operations.  Nett  Warrior 
consists  of  the  following: 

-  End  User  Device  (EUD),  a  commercial  off-the-shelf 
Samsung  Note  smartphone 

-  Government-furnished  Rifleman  Radio  (AN/PRC- 154 A) 
(Rifleman  Radio  is  a  separate  program  of  record,  with  a 
separate  article  later  in  this  Annual  Report.) 

-  Conformal  battery 

-  Connecting  cables 

-  Supporting  charging  equipment 

•  Periodic  Nett  Warrior  enhancements  integrate  improved 
commercial  EUD  technologies. 

•  The  Nett  Warrior  graphically  displays  the  location  of  an 
individual  leader,  other  leaders,  friendly  vehicles,  battlefield 


Mission 

•  Leaders  within  the  Brigade  Combat  Team  use  Nett  Warrior 
to  provide  improved  situational  awareness,  command  and 
control,  and  enhanced  communications. 

•  Combatant  Commanders  employ  Nett  Warrior-equipped 
infantry  and  cavalry  dismounted  leaders  as  part  of  a  Brigade 
Combat  Team  to  conduct  operations  (offensive,  defensive, 
stability,  and  defense  support  of  civil  authorities)  against 
conventional  or  unconventional  enemy  forces  in  all  types  of 
terrain  and  climate  conditions. 

Major  Contractors 

•  EUD:  Samsung  -  Seoul,  South  Korea 

•  Rifleman  Radio: 

-  General  Dynamics  C4  Systems  -  Scottsdale,  Arizona 

-  Thales  Communications  Inc.  -  Clarksburg,  Maryland 


Activity 

•  In  3QF  Y 14,  the  Assistant  Secretary  of  the  Army  for 
Acquisition,  Logistics,  and  Technology  authorized  an 
additional  LRIP  for  5,115  Nett  Warrior  systems  to  support 
fielding  to  Capability  Set  15-  and  16-fielded  units.  This 
additional  LRIP  allowed  the  program  manager  to  buy 
41  percent  of  the  Approved  Acquisition  Objective. 


•  ATEC  executed  an  IOT&E  in  two  phases: 

-  Phase  1,  conducted  by  a  mounted  cavalry  troop  during 
Network  Integration  Evaluation  14.2  at  Fort  Bliss,  Texas, 
in  May  2014. 

-  Phase  2,  conducted  by  a  dismounted  rifle  company  at 
Fort  Polk,  Louisiana,  in  November  2014. 
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•  DOT&E  intends  to  publish  an  IOT&E  report  in  2QFY15  to 

support  the  Full-Rate  Production  Decision  Review  in  3QFY15. 

•  The  Army  conducted  all  testing  in  accordance  with  the 

DOT&E-approved  Test  and  Evaluation  Master  Plan  and  test 

plans. 

Assessment 

IOT&E  Phase  1 

•  The  Nett  Warrior  was  effective  in  a  mounted  reconnaissance 
formation.  The  Nett  Warrior  improved  situational 
awareness  at  the  platoon  and  troop  level  and  enhanced 
pre-mission  planning  tasks,  land  navigation,  and  command 
and  control. 

•  Testing  revealed  interoperability  problems  between  Nett 
Warrior  and  the  Joint  Battle  Command  -  Platform  (JBC-P). 
The  Nett  Warrior  could  not  consistently  transmit  and 
receive  survivability  data  messages  in  the  variable  message 
formats  used  with  JBC-P.  Free  Text  messages  were  the 
most  common  type  of  command  and  control  messages  sent 
between  Nett  Warrior  and  JBC-P. 

•  Nett  Warrior  demonstrated  a  reliability  point  estimate  of 
445  operating  hours,  exceeding  the  threshold  and  objective 
requirement  of  148  and  291  hours,  respectively.  Nett 
Warrior  demonstrated  92  percent  operational  availability, 
exceeding  the  90  percent  requirement. 

•  The  Rifleman  Radio  battery  heated  to  temperatures 
exceeding  120  degrees  Fahrenheit.  Soldiers  experienced 
discomfort  from  hot  batteries.  The  Rifleman  Radio 
battery  life  did  not  meet  the  same  results  observed  in 
developmental  testing.  During  the  IOT&E,  battery 

life  was  2  to  6  hours  as  opposed  to  6  to  8  hours  during 
developmental  testing.  Post-test  analysis  showed  the 
battery  stopped  charging  when  it  reached  a  temperature 
of  1 18  degrees  Fahrenheit,  but  still  provided  power.  For 
further  information,  see  the  FY14  Annual  Report  article  on 
Rifleman  Radio. 

IOT&E  Phase  2 

•  The  Nett  Warrior  was  effective  in  light  infantry  platoon 
formations.  Use  of  the  Nett  Warrior  improved  situational 
awareness  at  the  platoon  level  and  continues  to  enhance 
pre-mission  planning  tasks,  land  navigation,  and  command 
and  control. 

•  The  infantry  company  was  hampered  by  the  short  range  of 
the  SRW  on  the  AN/PRC- 155  Manpack  radio  in  woodland 
terrain. 


•  The  Manpack  radio  connected  the  Nett  Warrior  systems 
of  the  leaders  between  platoons  to  each  other  and  to  the 
Nett  Warrior  systems  in  the  company  headquarters.  The 
infantry  company  did  not  have  complete  situational 
awareness  because  the  Manpack  radio  did  not  transmit 
position  location  information  in  a  consistent  manner.  SRW 
is  the  waveform  the  Army  has  selected  for  both  voice  and 
data  communications  at  these  tactical  echelons.  SRW 
demonstrates  a  shorter-range  vis-a-vis  the  legacy  Single 
Channel  Ground  and  Airborne  Radio  System  (SINCGARS) 
waveform  it  is  replacing  in  this  network.  This  result  is 

not  surprising  given  that  SRW  operates  at  a  much  higher 
frequency  than  does  SINCGARS,  as  higher  frequencies 
have  shorter  ranges  and  are  more  affected  by  terrain 
obstructions. 

•  The  test  unit  experienced  challenges  with  managing 
battery  charging  and  resupply  at  the  company  level.  The 
company  had  to  use  a  3-kilowatt  generator  from  battalion 
headquarters  to  charge  all  the  batteries  required  to  support 
the  company’s  Nett  Warrior  systems  and  Manpack 
radios.  The  generators  fielded  with  the  Nett  Warrior  were 
insufficient  to  support  the  company. 

•  The  Army  corrected  the  deficiency  of  the  radio  battery  not 
charging  in  high  temperatures  by  modifying  the  battery 
firmware  setting  to  allow  charging  at  higher  temperature 
limits,  up  to  the  high-temperature  exposure  limit  of  Military 
Standard-1472  (140  degrees  Fahrenheit  versus  118  degrees 
Fahrenheit). 

•  Nett  Warrior  is  suitable.  Nett  Warrior  demonstrated 
a  reliability  point  estimate  of  175.4  operating  hours, 
exceeding  the  threshold  requirement  of  148. 

•  DOT&E  will  publish  a  Nett  Warrior  IOT&E  report  in 
2QFY15. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed  all 

previous  recommendations. 

•  FY 14  Recommendation. 

1 .  The  Army  should  continue  to  improve  the  SRW  network 
and  associated  radios  to  increase  the  range  at  which 
Soldiers  and  leaders  can  use  Nett  Warrior. 
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Patriot  Advanced  Capability-3  (PAC-3) 


Executive  Summary 

•  The  Army  participated  in  the  second  and  final  Medium 
Extended  Air  Defense  System  (MEADS)  Flight  Test  (FT) 
using  three  Patriot  Advanced  Capability-3  (PAC-3)  Missile 
Segment  Enhancement  (MSE)  missiles  in  November  2013. 
The  missiles  successfully  engaged  a  short-range  ballistic 
missile  and  aircraft  target. 

•  The  DOD  decided  not  to  field  MEADS  and  concluded  U.S. 
involvement  in  the  design  and  development  phase  of  the 
MEADS  program  in  FY14. 

•  DOT&E  approved  the  latest  Patriot  System  Test  and 
Evaluation  Master  Plan  (TEMP)  in  December  2013. 

•  The  Defense  Acquisition  Executive  approved  the  PAC-3 
MSE  missile  to  enter  low-rate  initial  production  in  2QFY14. 

System 

•  Patriot  is  a  mobile  air  and  missile  defense  system  that 
counters  missile  and  aircraft  threats. 

•  The  system  includes  the  following: 

-  C-band  phased-array  radars  for  detecting,  tracking, 
classifying,  identifying,  and  discriminating  targets 

-  Battalion  and  battery  battle  management  elements 

-  Communications  Relay  Groups  and  Antenna  Mast 
Groups  for  communicating  between  battery  and  battalion 
assets 

-  A  mix  of  PAC-3  hit-to-kill  missiles  and  PAC-2  blast 
fragmentation  warhead  missiles  for  negating  missile  and 
aircraft  threats 

•  The  newest  version  of  the  Patriot  missile  is  the  PAC-3  MSE. 
The  PAC-3  MSE  missile  provides  increased  battlespace 
defense  capabilities  and  improved  lethality  over  prior 
configuration  Patriot  missiles. 

•  Earlier  versions  of  Patriot  missiles  include  the  Patriot 
Standard  missile,  the  PAC-2  Anti-Tactical  Missile,  the 
Guidance  Enhanced  Missile  (GEM)  family  (includes  the 
GEM-T  and  GEM-C  missile  variants  intended  to  counter 
tactical  ballistic  missiles  and  cruise  missiles,  respectively), 


the  PAC-3  (baseline),  and  the  PAC-3  Cost  Reduction  Initiative 
variant. 

•  The  DOD  intended  MEADS  to  replace  the  Patriot  system. 

The  DOD  decided  not  to  field  MEADS  and  concluded  U.S. 
involvement  in  the  design  and  development  phase  of  the 
MEADS  program  in  FY14. 

Mission 

Combatant  Commanders  use  the  Patriot  system  to  defend 
deployed  forces  and  critical  assets  from  missile  and  aircraft 
attack  and  to  defeat  enemy  surveillance  air  assets  (such  as 
unmanned  aerial  vehicles)  in  all  weather  conditions,  and  in 
natural  and  induced  environments. 

Major  Contractors 

•  Prime:  Raytheon  Company,  Integrated  Defense 
Systems  -  Tewksbury,  Massachusetts 

•  PAC-3  Interceptors:  Lockheed  Martin  Corporation,  Missile 
and  Fire  Control  -  Dallas,  Texas 


Activity 

•  During  MEADS  FT-2  in  November  2013  at  White  Sands 
Missile  Range,  New  Mexico,  two  MSE  interceptors  engaged 
a  short-range  ballistic  missile  target  and  a  third  MSE 
interceptor  engaged  a  full-scale  aircraft  target. 

•  DOT&E  approved  the  latest  Patriot  System  TEMP  in 
December  2013.  The  Army  conducted  all  testing  in 
accordance  with  the  DOT&E-approved  TEMP. 

•  The  PAC-3  MSE  missile  was  approved  to  enter  low-rate 
initial  production  in  2QFY14. 


•  The  Army  conducted  fragment  penetration  phenomenology 
tests  and  planned  high-explosive  initiation  tests  to  provide 
data  for  validation  of  PAC-3  lethality  models. 

•  In  September  2014,  the  Army  conducted  an  MSE 
developmental  lethality  test  against  a  ballistic  missile 
warhead  in  preparation  for  live  fire  lethality  testing  in  2015 
at  the  Holloman  AFB  high-speed  test  track. 

•  The  Army  planned  to  conduct  the  next  Patriot  operational 
test,  the  Post-Deployment  Build-8  IOT&E,  beginning 
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in  2015,  per  the  DOT&E-approved  TEMP.  However,  this 
date  is  likely  to  slip  to  at  least  2016  and  a  new  TEMP  is  under 
development  to  reflect  this  change.  The  IOT&E  will  provide 
information  to  support  the  Patriot  Full-Rate  Production  decision 
(including  the  MSE  missile). 

Assessment 

•  During  MEADS  FT-2,  MEADS  demonstrated  the  capability  to 
detect,  track,  engage,  intercept,  and  kill  both  a  tactical  ballistic 
missile  target  and  a  full-scale  aircraft  target  with  MSE  missiles. 

-  The  first  MSE  missile  in  the  ripple  method  of  fire 
intercepted  and  killed  the  ballistic  missile  target  at  the 
planned  altitude  and  range. 

-  The  second  MSE  missile  performed  nominally  throughout 
its  flight  and  properly  self-destructed  after  the  first  MSE 
intercepted  the  target. 

-  The  third  MSE  missile  intercepted  and  killed  the  full-scale 
aircraft  target  at  the  planned  altitude  and  range. 

•  The  MSE  developmental  lethality  test  in  September  2014  at  the 
Holloman  AFB  high-speed  test  track  met  the  lethality  objective. 

•  Patriot  ground  system  reliability  does  not  meet  the  threshold 
requirement  because  the  radar  performance  was  below 
threshold.  The  Project  Office  plans  to  replace  the  Patriot  radar, 
but  this  is  not  scheduled  to  occur  until  after  the  IOT&E.  The 
schedule  is  driving  entry  into  the  operational  test  prior  to 
implementing  the  fix,  and  resources/funding  are  not  available  to 
implement  the  fix.  Management  has  decided  to  accept  the  risk 
to  the  operational  test  assessment  of  not  implementing  the  fix. 

•  Patriot  training  remains  inadequate  to  prepare  operators  for 
complex  Patriot  engagements.  Resources/funding  are  not 
available  to  fix  this  issue  and  management  has  decided  to 
accept  the  risk  to  the  operational  test  assessment. 


Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  satisfactorily 

addressed  14  of  the  previous  23  recommendations.  The  Army 

should  continue  to  address  the  following  recommendations: 

1 .  Conduct  Patriot  air  and  missile  defense  testing  during 
joint  and  coalition  exercises  that  include  large  numbers 
of  different  aircraft  types,  sensors,  battle  management 
elements,  and  weapons  systems.  Conduct  Red  Team 
penetration  testing  during  these  exercises  to  test  Patriot 
cybersecurity. 

2.  Conduct  a  Patriot  flight  test  against  an  anti-radiation  missile 
target  to  validate  models  and  simulations. 

3.  Improve  Patriot  training  to  ensure  that  Patriot  operators  are 
prepared  to  use  the  system  in  combat. 

4.  Have  Patriot  participate  with  live  missiles  in 
Terminal  High- Altitude  Area  Defense  (THAAD)  flight 
testing  to  determine  Patriot-to-THAAD  interoperability  and 
the  capability  for  Patriot  to  intercept  tactical  ballistic  missile 
targets  that  are  not  intercepted  by  THAAD. 

5.  Collect  reliability  data  on  Patriot  systems  in  the  field  so  that 
the  Mean  Time  Between  Critical  Mission  Failure  can  be 
calculated. 

6.  Use  test  units  for  future  Patriot  operational  tests  that 
have  operationally  representative  distributions  in  Soldier 
proficiency. 

7.  Conduct  future  operational  flight  tests  with  unannounced 
target  launches  within  extended  launch  windows. 

8.  Improve  Patriot  radar  reliability. 

9.  Obtain  the  data  required  to  validate  GEM  interceptor  blast 
lethality  in  the  Lethality  Endgame  Simulation. 

•  FY 14  Recommendations.  None. 
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Precision  Guidance  Kit  (PGK) 


Executive  Summary 

•  The  Army  conducted  a  Limited  User  Test  of  the  Urgent 
Materiel  Release  version  of  the  Precision  Guidance  Kit  (PGK) 
in  2014  to  support  a  Full  Materiel  Release  (FMR).  The 
August  2014  FMR  decision  supported  the  use  of  the  Urgent 
Material  Release  PGK  in  training  and  future  contingency 
operations. 

•  In  September  2014,  DOT&E  published  an  Operational 
Assessment  report  in  support  of  FMR  of  the  Urgent  Material 
Release  PGK.  The  Urgent  Material  Release  PGK  met  the 
Army’s  key  requirements  for  accuracy,  reliability,  and 
compatibility  for  early  fielding. 

•  During  the  Program  of  Record  PGK  First  Article  Acceptance 
Tests  (FAAT),  the  project  manager  discovered  production 
quality  and  process  deficiencies  at  the  program’s  West  Virginia 
production  facility. 

•  The  Army  rescheduled  the  Program  of  Record  PGK  IOT &E 
from  2QFY14  to  3QFY15. 

•  The  contractor  moved  the  Program  of  Record  PGK  low-rate 
initial  production  (LRIP)  to  its  original  location  at  the 
contractor’s  production  facility  in  Minnesota.  LRIP  articles 
produced  in  Minnesota  will  undergo  FAAT  in  1QFY15. 

System 

•  The  PGK  is  a  combined  fuze  and  GPS  guidance  kit  that 
improves  the  ballistic  accuracy  of  the  current  stockpile  of 
high-explosive,  field  artillery  projectiles. 

•  The  Army  plans  to  develop  PGK  for  155  mm,  high-explosive 
projectiles  (M795  and  M549A1)  with  threshold  accuracy  of 
50  meters  Circular  Error  Probable  and  objective  accuracy  of 
30  meters  Circular  Error  Probable. 

•  The  PGK  will  operate  with  existing  and  developmental 
artillery  systems  that  have  digital  fire  control  systems  and 
inductive  fuze  setters  such  as  the  M777A2  Lightweight  Towed 
Flowitzer,  the  M109A6  Paladin  Self-Propelled  Howitzer,  and 
the  M109A7  Paladin  Integrated  Management  Self-Propelled 
Howitzer. 


Mission 

Field  Artillery  units  employ  PGK-fuzed  projectiles  in  support 
of  maneuver  units  to  provide  indirect  fires  with  50-meter 
accuracy.  PGK-fuzed  projectile  accuracy  allows  Field  Artillery 
units  to  fire  fewer  projectiles  to  achieve  comparable  effects  of 
conventionally-fuzed  artillery  ammunition. 

Major  Contractor 

Alliant-Techsystems  Advanced  Weapons 
Division  -  Plymouth,  Minnesota 


Activity 

•  PGK  entered  FY 14  with  two  active  program  tracks.  The  first 
track  is  the  Urgent  Material  Release  PGK,  which  focused 
on  meeting  an  Army-directed  requirement  for  deployed 
forces.  The  second  track  is  the  Program  of  Record  PGK  with 
full-rate  production  planned  for  4QFY14. 

Urgent  Material  Release  PGK 

•  In  October  2013,  the  Army  successfully  completed  the 
final  Lot  Acceptance  Test  and  Urgent  Material  Release 
PGK  production.  The  government  accepted  nearly 


3,135  Urgent  Material  Release  PGKs  for  the  Army  and 
Marines,  and  fielded  just  under  1,300  PGKs  to  deployed 
units  in  combat. 

•  In  1QFY14,  the  Army  announced  the  planned  end  of 
deployed  operations  and  the  return  of  approximately 
1,100  Urgent  Material  Release  PGKs  to  the  U.S. 

•  In  February  2014,  the  Army  executed  a  Limited  User  Test 
of  the  Urgent  Material  Release  PGK  at  Yuma  Proving 
Ground,  Arizona.  The  authorization  to  field  Urgent 
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Material  Release  PGKs  to  deployed  forces  restricted  the 
use  of  these  PGKs  to  support  Operation  Enduring  Freedom. 
The  Army  identified  the  need  to  obtain  an  FMR  for  the 
Urgent  Material  Release  PGKs  so  Soldiers  cotdd  use  them 
in  training  and  future  contingency  operations.  An  FMR 
for  the  Urgent  Material  Release  PGKs  required  a  favorable 
Operational  Test  Agency  Evaluation  Report. 

•  The  Army  approved  FMR  of  the  Urgent  Material  Release 
PGK  in  August  2014. 

•  DOT&E  published  an  Operational  Assessment  report 
supporting  the  FMR  of  the  Urgent  Material  Release  PGK  in 
September  2014. 

Program  of  Record  PGK 

•  Following  the  March  2013  Milestone  C  Decision  Review, 
which  approved  the  Program  of  Record  PGK  for  LRIP,  the 
Army  moved  the  PGK  production  line  from  Minnesota 

to  the  contractor’s  planned  production  facility  in  West 
Virginia. 

•  The  Program  of  Record  PGK  entered  FY14  with  a 
combined  PGK  and  Excalibur  Increment  lb  IOT&E 
scheduled  for  February  2014  at  Yuma  Proving  Ground, 
Arizona. 

•  In  November  2013,  the  contractor  delivered  the  First  Article 
sample  of  Program  of  Record  PGKs  from  the  West  Virginia 
production  line  to  the  government.  The  Army  rejected  the 
initial  First  Article  sample  because  14  failures  occurred  in 
80  of  the  PGKs  fired. 

•  The  Army  cancelled  Program  of  Record  PGK  participation 
in  the  scheduled  IOT&E  and  initiated  a  comprehensive 
Failure  Analysis  and  Corrective  Actions  investigation. 

The  Army  moved  LRIP  back  to  the  Minnesota  production 
facility  and  rescheduled  the  Program  of  Record  PGK 
IOT&E  for  3QFY15. 

•  In  April  20 1 4,  the  PGK  Milestone  Decision  Authority 
approved  an  updated  PGK  Acquisition  Program  Baseline 
Agreement.  This  agreement  identified  the  timeframe  for 
FMR  of  the  Program  of  Record  PGK  as  September  2015  to 
May  2016. 


•  The  PGK  Program  Management  Office  provided  DOT&E 
an  overview  of  a  testing  schedule  to  validate  corrective 
actions  and  ensure  readiness  of  PGK  to  enter  IOT&E  in 
3QFY15. 

•  The  Army  conducted  all  testing  in  accordance  with  the 
DOT&E-approved  Test  and  Evaluation  Master  Plan. 

Assessment 

Urgent  Material  Release  PGK 

•  In  operational  testing,  the  PGK  fuzes  demonstrated,  with 
confidence,  accuracy  results  well  within  the  accuracy 
requirement  of  a  50-meter  Circular  Error  Probable.  A 
median  radial  miss  distance  of  21.8  meters  demonstrated 
PGK  accuracy  exceeds  the  requirement  established  for 
obtaining  an  FMR  for  the  purpose  of  supporting  new 
equipment  training  and  fielding  to  the  Global  Response 
Force. 

Program  of  Record  PGK 

•  The  results  of  the  Army’s  comprehensive  Failure  Analysis 
and  Correction  Actions  attributed  failure  modes  to  quality 
and  process  deficiencies  at  the  West  Virginia  production 
plant  facility.  The  Army  has  not  validated  design  and 
production  process  changes  implemented  on  the  Minnesota 
production  line  in  government  testing.  FAAT,  which  is  a 
prerequisite  for  entry  into  IOT&E,  will  be  conducted  in 
1QFY15. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed 

previous  recommendations. 

•  FY 14  Recommendations.  The  Army  should: 

1 .  Continue  test  planning  to  validate  corrective  actions 
that  address  performance  and  production  shortfalls  and 
demonstrates  PGK’s  readiness  for  IOT&E. 

2.  Provide  an  updated  Test  and  Evaluation  Master  Plan  that 
documents  the  program’s  strategy  for  validating  corrective 
actions  to  address  performance  and  production  shortfalls. 
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Q-53  Counterfire  Target  Acquisition  Radar  System 


Executive  Summary 

•  In  April  and  May  2014,  the  Army  conducted  the  Q-53  radar 
IOT&E  at  Yuma  Proving  Ground,  Arizona.  Soldier  crews 
operated  four  Q-53  radars  during  four,  continuous  72-hour 
record  test  scenarios  observing  mortar,  artillery,  and  rocket 
fires.  Soldiers  conducted  counterfire  operations,  based  on  the 
tactical  scenario  presented. 

•  Based  on  IOT&E  results,  DOT&E  emerging  results  found  the 
Q-53  to  be  operationally  suitable,  not  operationally  effective, 
and  not  survivable.  The  Army  plans  to  conduct  additional 
operational  testing  and  DOT&E  is  working  with  them  to 
develop  the  scope  and  details  of  that  testing.  DOT&E  will 
then  re-evaluate  the  Q-53  based  on  the  changes  made  to  the 
radar  software  in  a  future  operational  test. 

•  The  Army  will  conduct  a  series  of  developmental  cyber  tests, 
to  include  threat  realistic  cyber  attacks,  and  the  tests  will 
culminate  in  an  operational  cyber  event. 

•  The  Army  Program  Executive  Officer  for  Missile  and  Space 
conducted  a  Q-53  radar  program  review  on  March  17,  2014, 
and  approved  the  procurement  of  Lot  4(13  systems). 

System 

•  The  Q-53  Counterfire  Target  Acquisition  Radar  System  is  a 
mobile  radar  system  designed  to  detect,  classify,  and  track 
projectiles  fired  from  mortar,  artillery,  and  rocket  systems 
using  a  90-degree  or  continuous  360-degree  sector  search. 

•  The  Army  intends  the  radar  to  provide  target  location  of  threat 
indirect  fire  systems  with  sufficient  timeliness  and  accuracy 
for  effective  counterfire. 

•  The  Q-53  is  designed  to  operate  with  the  Counter  -  Rocket, 
Artillery,  Mortar  system  and  the  future  Indirect  Fire  Protection 
Capability  system. 

•  The  Army  intends  to  field  the  Q-53  radar  to  the  target 
acquisition  platoons  in  Brigade  Combat  Teams  and  target 
acquisition  batteries  in  Fire  Brigades  to  replace  the  legacy 
AN/PQ-36  and  AN/TPQ-37  Firefinder  radars. 


•  The  Q-53  is  operated  by  a  crew  of  five  Soldiers  and 
transportable  by  C-17  aircraft.  Two  Family  of  Medium 
Tactical  Vehicle  trucks  provide  battlefield  mobility. 

•  The  Army  contracted  with  Lockheed  Martin  Missile  Systems 
and  Sensors  to  develop  and  field  38  Quick  Reaction  Capability 
radars  to  support  an  Urgent  Material  Release.  The  Army 
intends  to  produce  136  Program  of  Record  Q-53  radars. 

Mission 

Field  Artillery  units  employ  the  Q-53  radar  to  protect  friendly 
forces  by  determining  accurate  location  of  threat  rocket,  artillery, 
and  mortar  systems  for  defeat  with  counterfire  engagements. 

Air  Defense  Artillery  units  integrate  the  Q-53  radar  into  the 
Counter  -  Rocket,  Artillery,  Mortar  and  Indirect  Fire  Protection 
Capability  System  to  warn  friendly  forces  and  to  engage 
incoming  threat  indirect  fires. 

Major  Contractor 

Lockheed  Martin  Missile  Systems  and 
Sensors  -  Syracuse,  New  York 


Activity 

•  The  Army  began  training  in  October  2013  for  a  planned 
IOT&E  the  same  month.  Due  to  the  FY14  Federal  Government 
shutdown  and  lack  of  a  Defense  Appropriation,  travel 
restrictions  forced  the  Army  to  postpone  the  IOT&E  until 
April  2014. 

•  The  Army  completed  Developmental  Test  Phase  3  from 
December  2013  through  February  2014  at  Yuma  Proving 
Ground,  Arizona.  Developmental  testing  focused  on  reliability 
of  the  IOT&E  software. 


-  Civilian  crews  conducted  continuous,  72-hour  operations 
employing  the  radar  in  90-  and  360-degree  modes  with 
tactical  maneuver. 

-  After  each  72-hour  period,  and  as  the  schedule  permitted, 
crews  operated  the  radars  additional  hours  without 
movements.  The  radars  accumulated  1,033  hours  in 

six  test  cycles. 

-  The  Army  conducted  tests  characterizing  the  radar’s 
performance  in  the  90-degree  normal,  long-range 


Q-53  137 


F  Y  1  4  ARMY  PROGRAMS 


optimized  mode,  short-range  optimized  mode,  and 
360-degree  modes. 

•  The  Army  Program  Executive  Officer  for  Missile  and  Space 
conducted  a  Q-53  radar  program  review  on  March  17,  2014, 
and  approved  the  procurement  of  Lot  4(13  systems). 

•  In  April  and  May  2014,  the  Army  conducted  the  Q-53  radar 
IOT&E  at  Yuma  Proving  Ground,  Arizona,  in  accordance  with 
a  DOT&E-approved  Test  and  Evaluation  Master  Plan  and  test 
plan. 

-  Soldier  crews  operated  four  Q-53  radars  during  a  72-hour 
pilot  test  and  four  72-hour  record  test  scenarios  observing 
mortar,  artillery,  and  rocket  fires. 

-  The  radars  operated  in  90-  and  360-degree  modes 
throughout  the  IOT&E. 

-  Based  on  the  tactical  scenario  presented,  Soldiers  in 
the  counterfire  cell  managed  the  radars  and  conducted 
counterfire  operations. 

•  The  contractor  is  implementing  software  fixes  for  problems 
discovered  in  operational  testing.  The  Army  will  conduct  a 
series  of  developmental  cyber  tests,  to  include  threat  realistic 
cyber  attacks,  which  will  culminate  in  an  operational  cyber 
event.  DOT&E  is  working  with  the  Army  to  develop  the 
details  of  the  operational  cyber  test. 

•  The  Army  plans  to  conduct  IOT&E2  in  FY15. 

•  DOT&E  intends  to  publish  an  IOT&E  report  in  FY15  and  is 
currently  working  with  the  Army  to  develop  the  scope  and 
details  of  follow-on  testing. 

Assessment 

•  Based  on  IOT&E  results,  DOT&E  emerging  resilts  found  the 
Q-53  to  be  operationally  suitable,  not  operationally  effective, 
and  not  survivable.  DOT&E  will  re-evaluate  the  Q-53  changes 
made  to  the  radar  in  a  future  operational  test. 

-  The  Q-53  radar  is  operationally  suitable  because  the  radar 
was  available  to  complete  its  mission  99  percent  of  the 
time.  The  Army  requires  the  radar  to  be  operationally 
available  95  percent  of  the  time. 

-  The  Q-53  radar  did  not  meet  the  Army’s  reliability 
requirement.  During  the  IOT&E,  there  were  10  system 
aborts  during  1,152  operating  hours.  Based  on  these 
test  results,  there  is  an  80  percent  chance  the  radar  will 
average  greater  than  84  hours  between  system  aborts.  The 
Army  requires  the  system  to  have  an  80  percent  chance  of 
averaging  greater  than  185  hours  between  system  aborts. 
Soldiers  fixed  the  majority  of  the  failures,  which  were 
software  problems,  in  less  than  30  minutes,  resulting  in  a 
small  amount  of  downtime  and  high  availability. 

-  In  the  90-degree  mode  and  360-degree  mode,  the  radar  met 
accuracy  requirements  for  single-fired  artillery,  mortar,  and 
rocket  projectiles. 

The  counterfire  cell  was  able  to  manage  the  radars  and 
conduct  counterfire  missions  in  a  timely  fashion.  The 


Army  discovered  in  a  past  operational  test,  Soldiers  did 
not  receive  sufficient  training  in  the  counterfire  cell  to 
employ  the  radar  effectively.  Prior  to  the  IOT&E,  the 
Army  adjusted  the  counterfire  cell  training  and  replaced 
some  personnel  with  experienced  Advanced  Field  Artillery 
Tactical  Data  System  operators  in  the  counterfire  cell. 

-  The  radar  is  not  operationally  effective  because  of  the  large 
number  of  false  targets  reported  by  the  radar  in  the  IOT&E. 
While  in  the  90-degree  normal,  short-range  optimized 
mode  and  360-degree  modes,  the  radar  averaged  20,  32, 
and  7  false  targets  per  12  radiating  hours.  The  Army’s 
requirement  is  1  false  target  per  12  radiating  hours.  The 
Army  is  continuing  to  investigate  the  cause  of  the  high- 
false  location  rates.  The  contractor  developed  software 
fixes  for  false  target  problems  and  continues  to  assess  the 
software  changes. 

-  The  radar  is  not  operationally  effective  because  it 
had  difficulty  locating  volley-fired  projectiles  in  both 
90-  and  360-degree  modes.  Currently,  there  is  no  written 
requirement  for  the  radar  to  perform  against  volley-fired 
weapons.  Volley-fire  is  a  well-established  indirect  fire 
technique.  When  volley-fired  projectiles  were  located, 
the  radar  was  accurate  in  both  90-  and  360-degree  modes. 
DOT&E  added  the  volley-fire  mission  in  addition  to 

the  single-fire  mission  to  increase  the  test’s  operational 
realism;  however,  because  the  user  determined  there  is  no 
Q-53  requirement  for  volley-fired  projectiles,  the  Program 
Office  has  no  current  plans  to  address  the  issue. 

-  The  radar  is  not  survivable  against  a  persistent  cyber 
threat.  The  contractor  has  implemented  software  fixes 
for  problems  discovered  during  testing  and  the  Army  will 
conduct  an  operational  test  to  verify  those  fixes. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed 

one  of  the  three  previous  recommendations;  however,  the 

following  remain  outstanding: 

1 .  Confirm  and  characterize  suspected  radar-to-radar 
degradation  caused  by  violating  radar  contractor  positioning 
guidance.  Develop  and  test  techniques  to  overcome  radar 
degradation  if  contractor  positioning  guidance  is  confirmed. 

2.  Determine  if  there  is  a  valid  requirement  for  Q-53  radar 
performance  against  threat  munitions  fired  in  volleys. 

•  FY 14  Recommendations.  The  Army  should: 

1 .  Continue  to  investigate  the  cause  of  the  high-false  target 
rates  observed  in  the  IOT&E. 

2.  Continue  to  test  and  improve  the  radar’s  cyber  defenses. 

3.  Continue  to  improve  the  radar’s  capability  of  detecting 
volley-fired  projectiles  in  both  90-  and  360-degree  modes. 
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Rifleman  Radio 


Executive  Summary 

•  The  Army  tested  the  AN/PRC- 1 54A  Rifleman  Radio  as  part 
of  the  Nett  Warrior  IOT&E  Phases  1  and  2  in  accordance  with 
DOT&E-approved  test  plans: 

-  Phase  I  occurred  in  May  2014  during  Network  Integration 
Evaluation  (NIE)  14.2  at  Fort  Bliss,  Texas. 

-  Phase  II  occurred  in  November  2014  at 
Fort  Polk,  Louisiana. 

•  The  Army  plans  to  use  test  results  from  the  Nett  Warrior 
IOT&E  to  support  a  materiel  release  of  the  currently  fielded 
Low-Rate  Initial  Production  Rifleman  Radios. 

•  Voice  communications  are  good  when  the  radio  has 
line-of-sight  with  other  radios  in  open  terrain. 

•  At  the  Nett  Warrior  IOT&E  Phase  II  in  November  2014,  the 
AN/PRC- 154 A  Rifleman  Radio  supported  the  operations  of 
the  infantry  company’s  rifle  platoons  in  woodland  and  urban 
terrain. 

System 

•  The  Handheld,  Manpack,  and  Small  Form  Fit  (HMS)  program 
evolved  from  the  Joint  Tactical  Radio  System  program  and 
provides  software-programmable  digital  radios  to  support  the 
Army’s  tactical  communications  requirements.  The  two  radios 
that  comprise  the  HMS  program  are  the  AN/PRC- 154A 
Rifleman  Radio  and  the  AN/PRC-155  Manpack  radio 
(Manpack  radio  has  a  separate  article  earlier  in  this  Annual 
Report). 

•  The  Rifleman  Radio  is  a  handheld  networking  radio,  with 
National  Security  Agency  Type  1  encryption  suitable  for 
Secret  communications  and  data. 

•  In  addition  to  functioning  as  a  stand-alone,  handheld  radio, 
the  Army  intends  the  Rifleman  Radio  to  be  the  radio  used  as 
part  of  the  Nett  Warrior  program.  The  Army  is  developing  a 
vehicle-mounting  kit  for  the  radio. 

•  The  Rifleman  Radio  is  a  single-channel  radio  with  a 
commercial  GPS  receiver  to: 

-  Operate  at  various  transmission  frequencies  using  the 
Soldier  Radio  Waveform  (SRW),  which  enables  the  radios 


to  form  an  ad-hoc  data  and  voice  communications  network 
with  other  SRW-capable  radios 

-  Provide  5  watts  maximum  power  output  in  handheld,  and 
20  watts  maximum  power  output  in  vehicle  mount. 

-  Allow  Soldiers  to  transmit  position  location  information 
across  the  SRW  network.  The  position  location  capability 
of  the  Rifleman  Radio  is  disabled  when  connected  to 
Nett  Warrior. 

Mission 

Army  leaders  and  Soldiers  use  Rifleman  Radios  to  communicate 

and  create  networks  to  exchange  voice,  video,  and  data  using  the 

SRW  during  all  aspects  of  military  operations. 

Major  Contractors 

•  General  Dynamics,  C4  Systems  -  Scottsdale,  Arizona 

•  Thales  Communications,  Inc.  -  Clarksburg,  Maryland 


Activity 

•  The  Army  tested  the  AN/PRC- 1 54A  Rifleman  Radio  in 
accordance  with  DOT&E-approved  test  plans.  Testing  was 
conducted  as  part  of  Nett  Warrior  IOT&E  Phases  1  and  2: 

-  Phase  I  occurred  in  May  2014  during  NIE  14.2  at 
Fort  Bliss,  Texas. 

-  Phase  II  occurred  in  November  2014  at 
Fort  Polk,  Louisiana. 


•  The  Army  plans  to  use  test  results  from  the  Nett  Warrior 
IOT&E  to  support  a  material  release  of  the  currently  fielded 
Low-Rate  Initial  Production  Rifleman  Radio. 

•  The  Army  is  conducting  a  full  and  open  competition  of  the 
Rifleman  Radio  as  required  by  the  Defense  Acquisition 
Executive  for  IOT&E  and  the  Full-Rate  Production  decision. 
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•  The  Army  continues  preparation  for  a  future  Rifleman  Radio 
operational  test  on  the  Rifleman  Radio  variant  chosen  during 
the  full  and  open  competition. 

Assessment 

•  At  the  Nett  Warrior  IOT&E  Phase  1  in  May  2014  during 
NIE  14.2,  leaders  stated  that  voice  communications  were 
good  until  a  terrain  feature  blocked  line-of-sight. 

•  Soldiers  had  problems  with  the  radio  battery. 

-  The  battery  temperature  caused  first-degree  burns  and 
discomfort.  Sixty  percent  of  the  Soldiers  reported  that  the 
temperature  was  in  excess  of  120  degrees  Fahrenheit. 

-  The  battery  stopped  charging  with  the  excessive 
temperatures.  Post-test  analysis  discovered  that  by  design, 
the  battery  stops  charging  at  118  degrees  Fahrenheit,  but 
still  provided  power. 

•  The  radio  continues  to  demonstrate  the  following  suitability 
problems,  which  contributed  to  Soldiers  concluding  this  radio 
was  not  yet  acceptable  for  combat  in  its  current  Nett  Warrior 
configuration: 

-  Lack  of  a  display  screen  for  radio  status  as  Soldiers  change 
channels  or  prepare  radio  for  operation 

-  Battery  not  charging  at  temperatures  above  118  degrees 
Fahrenheit  and  rapid  battery  depletion 

•  At  the  Nett  Warrior  IOT&E  Phase  2  in  November  2014, 
the  Rifleman  Radio  supported  the  infantry  company’s  rifle 


platoons  as  they  operated  in  woodland  terrain  with  both  voice 
and  data  communications. 

•  The  Army  corrected  the  deficiency  of  the  radio  battery  not 
charging  in  high  temperatures  by  modifying  the  battery 
firmware  setting  to  allow  charging  at  higher  temperature 
limits,  up  to  the  high-temperature  exposure  limit  of 
Military  Standard-1472  (140  degrees  Fahrenheit  versus 
118  degrees  Fahrenheit). 

•  DOT&E  will  include  a  Rifleman  Radio  assessment  in  the 
Nett  Warrior  IOT&E  report,  which  will  be  published  in 
2QFY15. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army  addressed 
the  previous  recommendations. 

•  FY 14  Recommendations.  The  Army  should: 

1 .  Continue  to  complete  necessary  Rifleman  Radio 
documentation,  including  a  Test  and  Evaluation 
Master  Plan,  to  support  future  developmental  and 
operational  testing  as  part  of  the  full  and  open  competition. 

2.  Continue  to  execute  all  future  Rifleman  Radio  operational 
testing  to  support  Full-Rate  Production  decisions  with  the 
Nett  Warrior  system. 
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RQ-7BV2  Shadow  Tactical  Unmanned  Aircraft  System 

(TUAS) 


Executive  Summary 

•  The  Shadow  Tactical  Unmanned  Aircraft  System  (TUAS) 
program  completed  IOT&E  in  May  2002,  supporting  a 
Full-Rate  Production  decision  in  September  2002.  Since  that 
milestone,  the  Shadow  TUAS  Program  Office  has  fielded 
119  Shadow  systems.  The  Army  has  received  104;  the 
Marine  Corps,  13;  and  the  Australian  Army  (via  Foreign 
Military  Sales),  2.  The  Shadow  fleet  has  flown  over 
900,000  flight  hours. 

•  The  program  employs  a  block  upgrade  and  an  evolutionary 
acquisition  approach  and  has  to  date  produced  four  versions. 
The  four  versions  are  designated  the  RQ-7A,  RQ-7B, 
RQ-BV1,  and  the  RQ-7BV2.  The  latest  version,  the 
RQ-7BV2,  is  the  subject  of  this  report.  To  complement  this 
acquisition  approach,  the  T&E  Working  Integrated  Product 
Team  is  using  a  corresponding  test  strategy  as  part  of  a 
continuous  evaluation  as  the  system  receives  upgrades  in 
capability.  DOT&E  approved  the  Shadow  TUAS  Test  and 
Evaluation  Master  Plan  (TEMP)  update  in  May  2014. 

•  The  Army  conducted  the  Shadow  TUAS  FOT&E  at 
Fort  Bliss,  Texas,  and  the  White  Sands  Missile  Range, 

New  Mexico,  in  May  2014  during  the  Network  Integration 
Evaluation  (NIE)  14.2  and  the  Joint  Tactical  Exercise  Bold 
Quest. 

•  DOT&E  concludes  that  during  the  FOT&E,  the  RQ-7BV2 
Shadow  TU AS-equipped  unit  was  effective  at  employing 
the  system  and  demonstrated  it  is  capable  of  providing 
effective  support  to  combat  units;  the  Shadow  TUAS  is 
operationally  suitable,  but  not  reliable;  and  as  assessed 
after  the  2002  IOT&E,  the  Shadow  TUAS  is  not  survivable 
in  the  presence  of  an  air  threat  for  reconnaissance  and 
surveillance  missions.  Through  developmental  testing,  it 
has  been  determined  that  the  Shadow  TUAS  is  survivable 
in  the  presence  of  older,  less  capable  electronic  warfare 
threats,  but  further  testing  is  required  to  determine  Shadow’s 
performance  in  the  presence  of  the  latest  electronic  warfare 
threats. 

System 

•  The  Army  designed  the  Shadow  RQ-7BV2  to  provide 
coverage  to  a  brigade  area  of  interest  for  up  to  7  hours  at  a 
range  out  to  50  kilometers  from  the  launch  and  recovery  site. 
The  maximum  range  is  125  kilometers  (limited  by  datalink 
capability).  Operations  are  generally  conducted  from 
8,000  to  10,000  feet  above  ground  level  during  the  day  and 
6,000  to  8,000  feet  above  ground  level  at  night. 


1  •  RQ-7BV2  Aircraft  in  Flight 

2  -  RQ-7BV2  Aircraft  Being  Launched 

3  -  Inside  of  the  Univeral  Ground  Control  Station 

4  -  RQ-7BV2  Aircraft  Landing  Using  the  Arresting  Cable  Hook  System 

•  The  Laser  Range  Finder/Designator  provides  the  ground 
maneuver  Brigade  Commander  the  capability  to  conduct 
cooperative  HELLFIRE  missile  engagements. 

•  Shadow  RQ-7BV2  consists  of  the  following  major 
components: 

-  Four  small,  high-winged,  unmanned  aircraft,  each 
equipped  with  an  electro-optical  (EO)/Infrared  (IR) 
payload.  Two  of  the  four  EO/IR  payloads  are  equipped 
with  a  Laser  Range  Finder/Designator  capability.  The 
RQ-7BV2  aircraft  is  larger  than  the  RQ-7BV1  model 
primarily  through  an  extended  wing  modification  that 
increased  the  wingspan  of  the  aircraft  from  14  to  20.4  feet, 
added  additional  fuel  capacity,  and  increased  aircraft 
weight  from  375  to  460  pounds. 

-  Two  Ground  Control  Stations  designated  as  the  Universal 
Ground  Control  Station  (UGCS)  each  with  a  Universal 
Ground  Data  Terminal  (UGDT). 

-  One  Portable  Ground  Control  Station  (PGCS)  with  a 
Portable  Ground  Data  Terminal  (PGDT). 

-  An  integral  Single  Channel  Ground  and  Airborne  Radio 
System  (SINCGARS)  communications  relay  capability  on 
each  aircraft. 

-  Two  One-System  Remote  Video  Terminals  (OSRVT). 

•  The  Shadow  unit  is  a  platoon-size  organization  with 
27  personnel  authorized. 

•  The  aircraft  uses  a  hydraulic/pneumatic  launcher  and  is 
recovered  on  a  runway  using  the  Tactical  Automatic  Landing 
System.  An  arresting  cable/arresting  hook  system  shortens  the 
necessary  runway  landing  length. 


RQ-7BV2  Shadow  141 


F  Y  1  4  ARMY  PROGRAMS 


Mission  Major  Contractor 

The  Shadow  TUAS  platoon  is  to  provide  responsive  Textron  Systems  -  Hunt  Valley,  Maryland 

Reconnaissance,  Surveillance,  and  Security;  Cooperative  Attack; 

Battle  Damage  Assessment;  and  Communications  Relay  support 
to  the  brigade. 


Activity 

•  The  Shadow  TUAS  program  completed  IOT&E  in  May 
2002,  supporting  a  Full-Rate  Production  decision  in 
September  2002.  Since  that  milestone,  the  Shadow  TUAS 
Program  Office  has  fielded  119  Shadow  systems.  The  Army 
has  received  104;  the  Marine  Corps,  13;  and  the  Australian 
Army  (via  Foreign  Military  Sales),  2.  The  Shadow  fleet  has 
flown  over  900,000  flight  hours  with  over  755,000  of  those 
hours  being  flown  in  support  of  combat  operations.  As  of 
September  2014,  seven  deployed  systems  currently  support 
combat  operations. 

•  The  program  employs  a  block  upgrade  and  evolutionary 
acquisition  approach.  To  complement  this  approach, 
the  T&E  Worldng  Integrated  Product  Team  is  using  a 
corresponding  test  strategy  as  part  of  a  continuous  evaluation 
as  the  system  receives  upgrades  in  capability.  DOT&E 
approved  the  Shadow  TUAS  FOT&E  Operational  Test 
Plan  on  April  8,  2014,  and  the  Shadow  TEMP  update  on 
May  27,  2014. 

•  The  Army  conducted  the  Shadow  TUAS  FOT&E  at 
Fort  Bliss,  Texas,  and  the  White  Sands  Missile  Range, 

New  Mexico,  in  May  2014  during  NIE  14.2  and  the  Joint 
Tactical  Exercise  Bold  Quest,  in  accordance  with  the 
DOT&E-approved  TEMP  and  test  plan.  The  FOT&E 
enabled  the  evaluation  and  assessment  of  the  unit’s  ability  to 
employ  the  system  with  upgrades,  such  as  the  UGCS,  Tactical 
Common  Data  Link,  and  the  extended-wing  configuration  of 
the  aircraft  in  an  operational  environment.  The  test  provided 
the  opportunity  to  conduct  an  operational  assessment  of 

the  OSRVT-40  block  configuration  and  its  contributions  to 
the  supported  unit’s  situational  awareness.  The  test  was 
supported  by  two  AH-64D  Longbow  Aircraft  and  the  Brigade 
Combat  Team  fires  battalion  provided  indirect  fire  support 
during  call-for-fire  missions.  The  Shadow  test  unit  flew 
260  flight  hours  during  the  FOT&E. 

•  DOT&E  published  a  report  on  the  RQ-7BV2  Shadow 
TUAS  FOT&E  and  OSRVT  operational  assessment  in 
December  2014. 

Assessment 

•  During  FOT&E,  the  Shadow  TUAS-equipped  unit  was 
effective  at  employing  the  system  and  demonstrated  it  is 
capable  of  providing  effective  support  to  combat  units.  An 
example  of  this  support  occurred  during  one  two-day  mission 
during  FOT&E.  A  supported  Infantry  Battalion  utilized  the 
Shadow  TUAS  and  the  OSRVT  to  defeat  the  enemy  force  in 
detail. 


•  The  Shadow  TUAS  has  more  capability  and  functionality 
today  than  it  demonstrated  in  previous  operational  testing. 
Significant  increases  in  capability  demonstrated  in  the 
May  2014  FOT&E  include: 

-  The  ability  to  conduct  aircraft  operations  via  encrypted 
Tactical  Common  Data  Link 

-  Increased  aircraft  flight  endurance  due  to  the 
extended-wing  configuration 

•  The  Shadow  TUAS  is  operationally  suitable. 

-  The  Shadow  system  demonstrated  an  operational 
availability  of  88.6  percent  for  the  duration  of  FOT&E, 
exceeding  the  requirement  of  85  percent. 

-  During  FOT&E,  the  Shadow  TUAS  demonstrated  its 
ability  to  meet  its  normal  operational  tempo  requirement  of 
providing  16,  non-continuous  hours  of  on-station  time  in  a 
24-hour  period. 

-  The  RQ-7BV2-configured  Shadow  platoon  is  also  required 
to  support  surge  operations  consisting  of  72  hours  of 
continuous  time  on-station.  High  winds  throughout  the  test 
prevented  the  system  from  executing  a  72-hour  surge.  By 
design,  the  Army  provides  each  Shadow  platoon  with  the 
personnel  and  equipment  required  to  support  72-hour  surge 
operations.  In  an  attempt  to  assess  this  capability,  a  Monte 
Carlo  simulation  (repeated  random  sampling  of  reliability, 
availability,  and  maintainability  data  to  obtain  numerical 
results  for  the  amount  of  on-station  time  achieved  during 
any  72-hour  surge  period)  calculates  that  90  percent  of 

the  time,  Shadow  should  provide  64. 1  hours  of  coverage 
during  a  72-hour  surge  period  (an  89  percent  coverage 
capability). 

-  The  unit  achieved  the  operational  availability  requirement 
in  spite  of  failing  to  meet  its  reliability  requirement  due  to 
subsystem  redundancy. 

•  During  FOT&E,  the  Shadow  demonstrated  a  Mean  Time 
Between  System  Abort  (MTBSA)  point  estimate  of  8.7  hours 
versus  a  20-hour  MTBSA  requirement. 

•  The  unit  demonstrated  the  ability  to  conduct  cooperative 
HELLFIRE  missile  engagements  with  AH-64D  Longbow 
helicopter  crews.  The  Plug-In  Optical  Payload  (POP)  300D 
(Laser  Range  Finder/Designator  capable)  payload  continues  to 
support  cooperative  engagements  with  laser-guided  munitions. 

•  The  Median  Target  Location  Error  (TLE)  for  the  POP  300D 
payload  improved  from  74  meters  observed  during  the 

2010  Limited  User  Test  to  25  meters  observed  during  FOT&E. 
The  POP  300D  payload  TLE  requirement  is  50  meters. 
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•  During  FOT&E,  DOT&E  observed  that  in  order  to  more 
readily  anticipate  ground  commander  mission  support,  the 
Shadow  Platoon  Leader  sought  to  have  a  laser  designator 
capable  payload  on  every  mission.  As  reflected  in  the 
description  above,  the  system  is  fielded  with  two  of  the 
four  payloads  having  this  capability.  During  the  test, 

Shadow  maintenance  personnel  switched  payloads  from  one 
aircraft  to  another  2 1  times  in  order  to  have  the  most  laser 
designator  capable  flights.  This  frequent  swapping  of  payloads 
increases  the  probability  of  maintainer-induced  damage  to 

the  component  or  aircraft  as  well  as  increasing  maintainer 
workload. 

•  The  Shadow-equipped  unit  continues  to  demonstrate  the 
capability  of  conducting  effective  call  for  and  adjust  fire 
artillery  missions  via  the  Advanced  Field  Artillery  Tactical 
Data  System  electronic  messaging  system. 

•  The  Shadow  TUAS  communications  relay  capability  is 
provided  by  the  use  of  two  SINCGARS  radios  onboard 
the  aircraft,  one  in  each  wing  tip.  During  FOT&E  it  was 
observed: 

-  That  maintenance  personnel  on  the  ground  initially 
developed  and  continually  refined  pre-flight  procedures  for 
the  communications  relay  capability  to  make  sure  it  was 
operational  prior  to  launch. 

-  In  the  four  times  the  communications  relay  package 
was  employed  operationally  during  the  test,  it  worked 
successfully  three  times.  Shadow  operators  communicated 
with  AF1-64  aircraft  during  two  cooperative  engagement 
missions  and  on  one  other  occasion,  communicated  with 
their  forward-based  ground  unit  to  coordinate  aircraft 
movements  into  and  out  of  restricted  airspace.  In  the  one 
instance  when  the  communications  relay  package  did 

not  work,  Shadow  operators  could  hear  elements  of  the 
1-6  Infantry  unit  loud  and  clear,  but  Shadow  transmissions 
to  1-6  Infantry  were  broken  and  unreadable.  Unable 
to  communicate  by  voice  with  the  Shadow  unit,  the 
1-6  Infantry  commander  coordinated  mission  support  using 
chat. 

-  The  FOT&E  provided  no  data  to  evaluate  the 
communications  relay  capability  between  two  ground 
units.  Given  the  flat  terrain  and  the  digital  communications 
focus  of  the  NIE  exercise,  supported  units  did  not  need 

or  were  not  aware  of  the  Shadow  communications  relay 
capability.  Additional  testing  is  required  to  support  a 
complete  assessment  of  this  capability. 

•  The  Shadow  TUAS  demonstrated  the  capability  of  meeting  its 
8-hour  flight  endurance  requirement.  During  test,  its  longest 
flight  was  9.2  hours. 

•  The  UGCS  reduces  operator  work  load  when  compared 
with  the  legacy  One  System  Ground  Control  Station.  The 
checklist  steps  required  to  start  the  UGCS  and  launch  the 


Shadow  aircraft  have  decreased  from  514  to  456,  a  13  percent 
reduction.  Start-up  checklists  remain  time  consuming  and 
unforgiving  of  error  and  deviations.  Start-up  procedures, 
especially  for  the  configuration  of  the  encryption  datalink, 
added  to  workload  and  system  reliability  deficiencies. 

•  As  assessed  after  the  2002  IOT&E,  the  Shadow  TUAS  is  not 
survivable  in  the  presence  of  an  air  threat  for  reconnaissance 
and  surveillance  missions.  Since  the  2002  IOT&E,  there  has 
been  significant  development  and  proliferation  of  electronic 
warfare  threats  on  the  battlefield.  Through  developmental 
testing,  it  has  been  determined  that  the  Shadow  TUAS  is 
survivable  in  the  presence  of  older,  less  capable  electronic 
warfare  threats,  but  further  testing  is  required  to  detennine 
Shadow’s  performance  in  the  presence  of  the  latest  electronic 
warfare  threats.  Cybersecurity  testing  demonstrated  that 

the  Shadow  system  has  exploitable  vulnerabilities  that 
could  impact  Shadow  operations.  Cybersecurity  testing 
did  not  address  Detect,  React,  or  Restore  capabilities  of  the 
unit  equipped  with  the  Shadow  system  nor  did  it  test  the 
cybersecurity  of  the  UGCS  to  aircraft  control  datalinks. 
Further  details  can  be  found  in  the  classified  annex  to  the 
Shadow  Operational  Assessment  report. 

•  The  OSRVT-40  system  has  more  capability  and  functionality 
today  than  it  demonstrated  in  previous  operational  testing. 

Its  software  is  now  “plug  and  play,”  which  increases  user 
friendliness.  Additionally,  the  system  has  increased  range 
(mostly  through  the  use  of  the  Mobile  Directional  Antenna 
System)  and  it  supported  unit  Intelligence,  Surveillance,  and 
Reconnaissance  and  current  operations. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Army 
satisfactorily  addressed  the  four  recommendations  from  the 
FY10  DOT&E  Annual  Report.  There  was  no  annual  report 
written  for  this  system  in  FY11-13  due  to  lack  of  operational 
testing  during  that  time  period. 

•  FY 14  Recommendations.  The  Army  should: 

1.  Increase  the  number  of  POP-300D  payloads  issued  to  the 
Shadow  platoon  from  two  to  four  to  reduce  maintainer 
workload  and  increase  Shadow  platoon  flexibility. 

2.  Reduce  the  number  of  UGCS  checklist  steps  (and  automate 
the  process  to  the  greatest  degree  possible)  to  reduce  the 
amount  of  time  required  to  start-up  the  system  and  launch 
an  aircraft. 

3 .  Conduct  continued  developmental  testing  to  further 
characterize  Electronic  Warfare  threats  against  the  Shadow. 

4.  Conduct  additional  cybersecurity  testing  to  fully  assess 
detect,  react,  and  restore  system  capabilities  of  units 
equipped  with  the  Shadow  TUAS.  Conduct  cybersecurity 
testing  on  the  UGCS-to-aircraft  control  datalinks. 
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Warfighter  Information  Network  -  Tactical  (WIN-T) 


Executive  Summary 

•  In  September  2013,  the  Defense  Acquisition  Executive  (DAE) 
conducted  a  Warfighter  Information  Network  -  Tactical 
(WIN-T)  Increment  2  Full-Rate  Production  (FRP)  Defense 
Acquisition  Board  (DAB)  based  upon  the  May  2013  WIN-T 
Increment  2  FOT&E.  DOT&E  presented  an  FOT&E  report 
that  assessed: 

-  Most  WIN-T  Increment  2  configuration  items  were 
operationally  effective.  The  Soldier  Network  Extension 
(SNE),  Tactical  Radio  -  Tower  (TR-T),  and  High  Band 
Networking  Waveform  (HNW)  were  not  operationally 
effective. 

-  Most  WIN-T  Increment  2  configuration  items  were 
operationally  suitable.  The  SNE  and  Point  of  Presence 
(PoP)  were  not  operationally  suitable. 

-  WIN-T  Increment  2  demonstrated  improvement  in 
survivability,  but  requires  further  improvement  in 
cybersecurity. 

•  Asa  result  of  the  September  2013  DAB,  the  DAE  published  an 
Acquisition  Decision  Memorandum  (ADM)  that: 

-  Authorized  the  acceptance  of  the  2012  low -rate  initial 
production  (LR1P)  and  procurement  of  another  LRIP 
without  SNEs. 

-  Directed  the  Army  to  reduce  SNE  and  PoP  complexity, 
improve  PoP  reliability,  fix  survivability,  correct 
deficiencies  discovered  during  the  FOT&E,  and 
demonstrate  these  improvements  in  a  second  FOT&E. 

•  During  FY14,  the  Army  conducted  two  WIN-T  Increment  2 
developmental  tests,  which  demonstrated  improvements  in 
SNE  and  PoP  operations,  but  did  not  provide  confidence  that 
the  SNE  and  PoP  would  meet  reliability  requirements  during 
the  planned  second  FOT&E.  The  Army  has  again  reduced 
its  approved  reliability  requirements  for  selected  WIN-T 
components,  including  the  SNE,  to  levels  now  likely  to  be 
consistent  with  demonstrated  performance. 

•  The  Army  conducted  a  second  WIN-T  Increment  2  FOT&E 
(FOT&E2)  during  Network  Integration  Evaluation  (NIE)  15.1 
from  October  to  November  2014. 

•  The  program  updated  the  WIN-T  Increment  2  Test  and 
Evaluation  Master  Plan  (TEMP)  to  support  the  second  FOT&E 
and  completed  a  WIN-T  Increment  2  TEMP  annex  that  outlines 
the  remaining  WIN-T  Increment  3  test  activities. 

System 

•  The  Army  designed  the  WIN-T  as  a  three-tiered 
communications  architecture  (space,  terrestrial,  and  airborne) 
to  serve  as  the  Army’s  high-speed  and  high-capacity  tactical 
communications  network. 

•  The  Army  intends  WIN-T  to  provide  reliable,  secure,  and 
seamless  communications  for  units  operating  at  theater  level 
and  below. 


Point  of  Presence 


M-ATV  Soldier  Network  Extension 


Stryker  Point  of  Presence 

1  -  Net-Centric  Waveform 

Antenna 

2  -  High-Band  Networking 

Waveform  Antenna 

M-ATV  -  Mine  Resistant  Ambush  Protected 
(MRAP)  All-Terrain  Vehicle  (M-ATV) 


Tactical  Comms  Node 


•  The  WIN-T  program  consists  of  four  increments.  In 
May  2014,  the  DAE  approved  the  Army’s  request  to  stop 
development  of  the  Increment  3  aerial  tier  of  networked 
airborne  communications  relays  and  limit  Increment  3  to 
network  management  and  satellite  waveform  improvements. 
The  Army  intends  to  increase  procurement  of  WIN-T 
Increment  2  configuration  items  to  satisfy  capability 
set  requirements  previously  planned  for  Increment  3. 
Increment  4  is  currently  not  funded. 

-  Increment  1:  “Networking  At-the-Halt”  enables  the 
exchange  of  voice,  video,  data,  and  imagery  throughout 
the  tactical  battlefield  using  a  Ku-  and  Ka-satellite-based 
network.  The  Army  has  fielded  WIN-T  Increment  1  to  its 
operational  forces. 

-  Increment  2:  “Initial  Networking  On-the-Move”  provides 
command  and  control  on-the-move  down  to  the  company 
level  for  maneuver  brigades  and  implements  an  improved 
network  security  architecture.  WIN-T  Increment  2 
supports  on-the-move  communications  for  commanders 
with  the  addition  of  the  PoP  and  the  SNE  and  provides 

a  mobile  network  infrastructure  with  the  Tactical 
Communications  Node. 

-  Increment  3:  “Full  Networking  On-the-Move”  provides 
full  mobility  command  and  control  for  all  Army  field 
commanders,  from  theater  to  company  level.  Network 
reliability  and  robustness  are  enhanced  with  the  addition 
of  the  air  tier  transport  layer,  which  consists  of  networked 
airborne  communications  relays. 


WIN-T 
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-  Increment  4:  “Protected  Satellite  Communications  On-the 
Move”  includes  access  to  the  next  generation  of  protected 
communications  satellites  while  retaining  all  previous 
on-the-move  capabilities. 

Mission 

Commanders  at  theater  level  and  below  will  use  WIN-T  to: 

•  Integrate  satellite-based  communications  capabilities  into 
an  everything-over-lnternet  Protocol  network  to  provide 
connectivity,  while  stationary,  across  an  extended,  non-linear 
battlefield  and  at  remote  locations  (Increment  1) 


•  Provide  division  and  below  maneuver  commanders  with 
mobile  communications  capabilities  to  support  initial 
command  and  control  on-the-move  (Increment  2) 

•  Provide  all  maneuver  commanders  with  mobile 
communications  capabilities  to  support  full  command 
and  control  on-the-move,  including  the  airborne  relay  and 
protected  satellite  communications  (Increments  3  and  4) 

Major  Contractor 

General  Dynamics,  C4  Systems  -  Taunton,  Massachusetts 


Activity 

•  In  response  to  a  September  2012  DAB,  the  Army  conducted 
the  WIN-T  Increment  2  FOT&E  in  May  2013  as  part  of 
NIE  13.2.  The  test  employed  the  2nd  Brigade,  1st  Armored 
Division  under  operationally  realistic  conditions  at  Fort  Bliss, 
Texas,  and  White  Sands  Missile  Range,  New  Mexico. 

•  In  September  2013,  DOT&E  published  a  WIN-T  Increment  2 
FOT&E  report  to  support  a  September  2013  FRP  DAB.  This 
report  assessed: 

-  Most  WIN-T  Increment  2  configuration  items  were 
operationally  effective.  The  SNE,  TR-T,  and  HNW  were 
not  operationally  effective. 

-  Most  WIN-T  Increment  2  configuration  items  were 
operationally  suitable.  The  SNE  and  PoP  were  not 
operationally  suitable. 

-  WIN-T  Increment  2  demonstrated  improvement  in 
survivability,  but  requires  further  improvement  in 
cybersecurity. 

•  In  September  2013,  the  DAE  conducted  a  WIN-T  Increment  2 
FRP  DAB.  The  resulting  ADM: 

-  Authorized  the  acceptance  of  the  2012  LRIP  and  the 
procurement  of  another  LRIP  without  SNEs. 

-  Directed  additional  developmental  testing  followed  by 
FOT&E2  to  confirm  correction  of  deficiencies  noted 
during  FOT&E  and  to  demonstrate — 

■  The  PoP  meets  threshold  reliability  requirements. 

■  Significant  reductions  in  the  complexity  of  start-up, 
troubleshooting,  and  shutdown  procedures  for  the  SNE 
and  PoP. 

■  Significant  reductions  in  the  complexity  of  the  SNE 
Combat  Network  Radio  (CNR)  Gateway  operations. 

•  In  May  2014,  the  DAE  approved  the  Army’s  proposed 
restructure  of  the  WIN-T  Increment  3  program  to  defer  the 
air  tier,  the  Joint  Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance,  and  Reconnaissance 
(JC41SR)  radio,  and  cease  all  efforts  associated  with 
development  of  unique  hardware  items.  The  program  will 
complete  development  of  the  Net  Centric  Waveform  10.X 
software  upgrade,  and  network  operations  software,  and 
transition  these  capabilities  to  WIN-T  Increments  1  and  2. 

The  program  will  demonstrate  the  HNW  version  3.0  waveform 


(supports  the  JC41SR  radio)  and  then  store  the  waveform  in 
the  DOD  Waveform  Repository. 

•  The  Army  redesigned  the  user  interface  for  the  SNE,  PoP, 
and  CNR  Gateway  to  reduce  complexity  and  exhibited  these 
improvements  at  a  demonstration  event  at  Aberdeen  Proving 
Ground,  Maryland. 

•  During  2-3QFY14,  the  Army  conducted  two  developmental 
tests  at  Aberdeen  Proving  Ground,  Maryland,  and  Fort  Bliss, 
Texas.  These  events  were  designed  to  verify  the  corrective 
actions  to  address  SNE  and  PoP  usability  and  reliability 
prior  to  FOT&E2.  The  second  developmental  test  employed 
Soldiers  operating  SNE  and  PoP  variants  of  Mine  Resistant 
Ambush  Protected  and  Stryker  vehicles  under  a  rigorous  test 
scenario. 

•  Per  ADM  direction,  the  program  updated  the  WIN-T 
Increment  2  TEMP  to  support  FOT&E2  and  completed  a 
WIN-T  Increment  2  TEMP  annex  that  outlines  the  remaining 
WIN-T  Increment  3  test  activities. 

•  On  October  9,  2014,  the  Army  approved  a  revised  WIN-T 
Increment  2  reliability  requirement,  Mean  Time  Between 
Essential  Function  Failure  (MTBEFF),  which  reduced  the 
reliability  requirements  for  the  majority  of  WIN-T  Increment  2 
configuration  items.  The  Army  modified  the  requirement  by 
adjusting  WIN-T  system  utilization  rates  within  a  72-hour 
mission  duration  based  upon  previous  operational  testing. 

This  adjustment  resulted  in  the  following  changes  to  MTBEFF 
requirements: 

-  SNE  decreased  from  250  to  1 84  hours. 

-  PoP  decreased  from  182  to  144  hours. 

-  Vehicle  Wireless  Package  increased  from  45  to  97  hours. 

-  Tactical  Communications  Node  decreased  from  3 14  to 
303  hours. 

-  Network  Operations  and  Security  Center  decreased  from 
218  to  216  hours. 

-  TR-T  decreased  from  146  to  142  hours. 

•  Since  the  original  2008  WIN-T  Increment  2  Capability 
Production  Document  MTBEFF  requirement  of  900  hours  for 
all  systems  except  SNE  and  TR-T  (300  hours),  the  Army  has 
reduced  its  reliability  requirements  three  times  in  advance  of 
operational  testing  based  upon  the  following  modifications. 
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-  Prior  to  the  May  2012  IOT&E,  reduced  mission  duration 
from  96  to  72  hours. 

-  Prior  to  the  May  2013  FOT&E,  reduced  probability  of 
mission  completion  from  90  to  80  percent. 

-  Prior  to  the  October  2014  FOT&E2,  adjusted  system 
utilization  rates. 

•  From  October  to  November  2014,  the  Army  conducted 
the  WIN-T  Increment  2  FOT&E2  during  NIE  15.1 

at  Fort  Bliss,  Texas,  and  White  Sands  Missile  Range, 

New  Mexico.  FOT&E2  included  operational  testing  to  assess 
the  integration  of  SNE  and  PoP  equipment  into  Stryker  combat 
vehicles. 

Assessment 

•  The  Army’s  demonstration  of  an  improved  user  interface 
for  the  SNE,  PoP,  and  CNR  gateway  presented  an  intuitive, 
easy-to-use  method  for  operations  and  troubleshooting. 

•  During  the  two  developmental  test  events,  WIN-T  Increment  2 
demonstrated: 

-  Improved  SNE  and  PoP  usability,  which  Soldiers  viewed  as 
easier  to  operate. 

-  Reduced  SNE  and  PoP  startup  and  shutdown  time. 

-  Simplified  SNE  and  PoP  troubleshooting. 

-  Improved  the  CNR  Gateway  interface,  which  simplifies 
operations  and  troubleshooting. 

-  Although  improved  since  FOT&E,  reliability  results  do  not 
provide  confidence  of  the  SNE  and  PoP  meeting  reliability 
requirements  during  FOT&E2. 

•  The  program  validated  fixes  for  all  identified  failure  modes 
during  the  FOT&E  and  the  first  developmental  test,  and 
intended  to  demonstrate  these  fixes  during  FOT&E2.  The 
assessment  is  ongoing. 


•  The  Army  continues  to  test  WIN-T  Increment  2  in  a  brigade 
configuration  and  has  not  conducted  an  operational  test  on 

a  complete  division.  With  each  release  of  software,  WIN-T 
Increment  2  has  experienced  new  failure  modes  during 
operational  test. 

•  WIN-T  Increment  2  continues  to  demonstrate  cybersecurity 
vulnerabilities.  This  is  a  complex  challenge  for  the  Army  since 
WIN-T  is  dependent  upon  the  cyber  defense  capabilities  of  all 
systems  connected  to  the  network,  for  example.  Distributed 
Common  Ground  System  -  Army  and  Joint  Warning  and 
Reporting  Network. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  program  addressed 
one  of  three  previous  recommendations.  They  still  need  to 
improve  the  range  of  HN W  and  demonstrate  recommended 
improvements  during  the  WIN-T  Increment  2  FOT&E2, 
conducted  during  NIE  15.1.  The  assessment  is  ongoing. 

•  FY 14  Recommendations.  The  Army  should: 

1.  Assess  the  results  of  the  WIN-T  Increment  2  FOT&E2  and 
conduct  an  operational  test  to  demonstrate  correction  of 
deficiencies  noted  during  test. 

2.  Conduct  an  operational  test  to  assess  the  Net  Centric 
Waveform  and  network  operations  capabilities  provided  by 
WIN-T  Increment  3. 

3.  Conduct  an  operational  test  to  assess  the  future  integration 
of  WIN-T  Increment  2  into  combat  vehicles. 

4.  Improve  HNW  and  TR-T  to  gain  better  transmission  range 
from  the  radio  and  increase  the  number  of  TR-Ts  available 
to  support  units  in  dispersed  operations. 
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Aegis  Modernization  Program 


Executive  Summary 

•  The  Navy  is  modernizing  the  Aegis  Weapon  System  (AWS) 
installed  on  Baseline  3  USS  Ticonderoga  (CG  47)  class 
cruisers  and  the  Flight  I  USS  Arleigh  Burke  (DDG  51) 
destroyers  to  the  AWS  Advanced  Capability  Build  2012 
(Baseline  9A  and  9C,  respectively).  New  construction  DDGs, 
beginning  with  USS  John  Finn  (DDG  113),  will  be  equipped 
with  Baseline  9C  as  well. 

•  In  September  2014,  the  Navy  completed  cruiser  integrated  air 
defense  and  Undersea  Warfare  developmental  and  operational 
testing  on  USS  Normandy  and  USS  Chancellorsville.  Data 
from  these  tests  will  supplement  data  from  the  dedicated 
operational  test  in  FY15. 

•  During  developmental  testing,  a  BQM-74E  anti-ship  cruise 
missile  (ASCM)  target  drone  struck  USS  Chancellorsville. 

The  time  required  to  repair  the  ship  delayed  the  start  of  cruiser 
operational  testing,  originally  planned  for  4QFY14,  until 
2QFY15. 

•  The  lack  of  an  adequate  modeling  and  simulation  (M&S) 
suite  of  the  Aegis  Combat  System,  as  well  as  the  lack  of  an 
Aegis-equipped  Self-Defense  Test  Ship  (SDTS)  where  the 
ship’s  full  self-defense  kill  chain  can  be  tested,  precludes 
assessment  of  the  Baseline  9  Probability  of  Raid  Annihilation 
(PRA)  requirement. 

•  The  Navy  will  not  fully  assess  Aegis  Integrated  Air  and 
Missile  Defense  (IAMD)  until  a  validated  M&S  test  bed 
is  developed  and  validated.  The  test  bed  is  planned  to  be 
available  by  2020,  but  there  is  no  agreed  upon  strategy  to 
validate  the  model  to  support  assessment  of  the  close-in, 
self-defense  battle  space.  A  limited  IAMD  assessment  will 
be  made  during  Baseline  9C  operational  testing  on  DDGs. 

One  live  firing  event  is  planned  to  include  live  firing  of 
Standard  Missile-2  (SM-2)  and  SM-3  missiles  against  threat 
representative  targets  in  an  IAMD  engagement. 

•  During  developmental  testing  in  June  2014,  the  Navy 
successfully  conducted  three,  at-sea  live  fire  tests  of  the 
Naval  Integrated  Fire  Control  -  Counter  Air  From-the-Sea 
(NIFC-CA  FTS)  Increment  I  capability.  The  Navy  will  field 
the  NIFC-CA  FTS  Increment  I  capability  when  it  deploys  the 
first  E-2D  and  Aegis  Baseline  9-equipped  Carrier  Strike  Group 
in  FY15.  NIFC-CAFTS  Increment  I  has  demonstrated  a  basic 
capability,  but  its  effectiveness  under  operationally  realistic 
conditions  is  undetermined. 

System 

•  The  Navy’s  Aegis  Modernization  program  provides  updated 
technology  and  systems  for  existing  Aegis  guided  missile 
cruisers  (CG  47)  and  destroyers  (DDG  51).  This  planned, 
phased  program  provides  similar  technology  and  systems  for 
new  construction  destroyers. 


. 

? 


•  The  AWS,  carried  on  DDG  5 1  guided  missile  destroyers 
and  CG  47  guided  missile  cruisers,  integrates  the  following 
components: 

-  AWS  AN/SPY-1  three-dimensional  (range,  altitude,  and 
azimuth)  multi-function  radar 

-  AN/SQQ-89  Undersea  Warfare  suite  that  includes  the 
AN/SQS-53  sonar,  SQR-19  passive  towed  sonar  array 
(DDGs  51  through  78,  CGs  52  through  73),  and  the 
SH-60B  or  MH-60R  Helicopter  (DDGs  79  Flight  IIA 
and  newer  have  a  hangar  to  allow  the  ship  to  carry  and 
maintain  its  own  helicopter) 

-  Close-In  Weapon  System 

-  Five-inch  diameter  gun 

-  Harpoon  ASCMs  (DDGs  5 1  through  78,  CGs  52 
through  73) 

-  Vertical  Launch  System  that  can  launch  Tomahawk 
land-attack  missiles,  Standard  surface-to-air  missiles, 
Evolved  SeaSparrow  Missiles,  and  Vertical  Launch 
Anti-Submarine  Rocket  missiles 

•  The  AWS  on  Baseline  3  USS  Ticonderoga  (CG  47)  class 
cruisers  and  Flight  1  USS  Arleigh  Burke  destroyers  is  being 
upgraded  to  Baseline  9A  and  9C,  respectfully.  Baseline  9  will 
provide  the  following  new  capabilities: 

-  Full  SM-6  integration 

-  IAMD  to  include  simultaneous  Air  Defense  and  Ballistic 
Missile  Defense  missions  on  Aegis  destroyers  equipped 
with  the  new  Multi-Mission  Signal  Processor 

-  NIFC-CAFTS  capability 

-  Starting  with  USS  John  Finn  (DDG  113),  the  AWS  on 
new  construction  Aegis  guided  missile  destroyers  is 
Baseline  9C 
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Mission 

The  Joint  Force  Commander/Strike  Group  Commander 
employs  AWS-equipped  DDG  51  guided  missile  destroyers  and 
CG  47-guided  missile  cruisers  to  conduct: 

•  Area  and  self-defense  Anti-Air  Warfare  in  defense  of  the  Strike 
Group 

•  Anti- Surface  Warfare  and  Anti-Submarine  Warfare 

•  Strike  Warfare  when  armed  with  Tomahawk  missiles 

•  Offensive  and  defensive  warfare  operations  simultaneously 

•  Independent  operations  or  with  Carrier  or  Expeditionary  Strike 
Groups,  as  well  as  with  other  joint  or  coalition  partners 


Major  Contractors 

•  General  Dynamics  Marine  Systems  Bath  Iron 
Works  -  Bath,  Maine 

•  Huntington  Ingalls  Industries  (formerly  Northrop  Grumman 
Shipbuilding)  -  Pascagoula,  Mississippi 

•  Lockheed  Martin  Maritime  Systems  and 
Sensors  -  Moorestown,  New  Jersey 


Activity 

•  In  June  2014,  the  Navy  successfully  conducted 

three  N1FC-CAFTS  Increment  1  engineering  demonstration 
tests  on  USS  John  Paul  Jones  (DDG  53).  Although  not  part  of 
a  DOT&E-approved  test  plan,  DOT&E  observed  and  collected 
performance  data  on  the  NIFC-CA  FTS  Increment  I  capability. 

•  The  Navy  successfully  conducted  the  Baseline  9A  Cruiser 
integrated  air  defense  and  Undersea  Warfare  developmental 
and  operational  testing  on  USS  Normandy  (CG  60)  and 
USS  Chancellorsville  (CG  62)  in  accordance  with  the 
DOT&E-approved  test  plan.  Data  from  these  tests  will 
supplement  data  from  the  dedicated  operational  test  in  FY15. 

•  During  developmental  testing,  a  BQM-74E  ASCM  target 
drone  struck  USS  Chancellorsville  (CG  62).  The  time 
required  to  repair  the  ship  delayed  that  start  of  operational 
testing,  originally  planned  for  4QFY14,  until  2QFY15. 

•  Aegis  Destroyer  Baseline  9C  dedicated  operational  testing  is 
scheduled  for  FY16. 

•  The  Navy  conducted  Baseline  9A  cybersecurity  operational 
testing  onboard  USS  Chancellorsville  (CG  62)  in 
November  and  December  2014. 

Assessment 

•  Baseline  9A  and  9C  testing  completed  to  date  is  not  sufficient 
to  support  an  assessment  of  operational  effectiveness  or 
suitability  before  the  first  ship  deploys  in  FY15.  Operational 
testing  is  planned  to  continue  throughout  FY15/16.  Upon  the 
decision  to  deploy  Baseline  9A  in  FY15,  DOT&E  will  submit 
an  Early  Fielding  Report. 

•  The  Navy  will  not  fully  assess  Aegis  IAMD  until  a  validated 
M&S  test  bed  is  developed  and  validated.  The  test  bed 

is  planned  to  be  available  by  2020,  but  there  is  no  agreed 
upon  strategy  to  validate  the  model  to  support  assessment 
of  the  close-in,  self-defense  battle  space.  A  limited  IAMD 
assessment  will  be  made  during  Baseline  9C  operational 
testing  on  DDGs.  One  live  firing  event  managed  by  the 
Missile  Defense  Agency  is  planned  to  include  live  firing  of 
SM-2  and  SM-3  missiles  against  threat  representative  targets 
in  an  IAMD  engagement  in  FY15. 

•  As  appropriate,  and  until  the  full  capability  may  be 
operationally  tested,  DOT&E  will  provide  periodic  capability 


assessments  to  inform  Navy  and  OSD  leadership,  as  well  as 
Congress,  on  the  progress  of  test  and  evaluation  of  the  IAMD 
mission  area. 

•  Until  an  Aegis-equipped  SDTS  is  available  for  testing,  it  is 
neither  possible  to  characterize  the  self-defense  capabilities 
of  the  Aegis  cruisers  and  destroyers,  nor  possible  to  accredit 
an  M&S  suite  to  determine  if  the  ships  satisfy  their  PRA 
requirements. 

•  The  Navy’s  FY14  NIFC-CA  FTS  Increment  I  events  were 
sufficient  to  demonstrate  basic  capability;  however,  these 
demonstrations  were  not  conducted  under  operationally 
realistic  conditions  or  against  aerial  targets  representative  of 
modern  threats.  Additionally,  the  executed  scenarios  were 
not  sufficiently  challenging  to  demonstrate  the  NIFC-CA 
requirements  defined  in  the  Navy’s  September  2012  NIFC-CA 
Testing  Capability  Definition  Letter. 

•  Combined  Aegis  Baseline  9  and  SM-6  FOT&E  test  events  to 
date  have  been  successful  with  no  integration  issues  revealed. 
The  Navy  plans  to  conduct  six  SM-6/Baseline  9  test  flights  in 
FY15. 

•  The  Navy’s  Aegis  Baseline  9A  cybersecurity  testing  revealed 
significant  problems,  which  are  classified.  The  nature  of  these 
problems  is  such  that  they  coidd  pose  significant  risk  to  the 
cybersecurity  for  the  FY15  deployment. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  not 
addressed  two  of  the  three  previous  recommendations.  The 
Navy  still  needs  to: 

1.  Continue  to  improve  Aegis  ships’  capability  to  counter 
high-speed  surface  threats  in  littoral  waters. 

2.  Synchronize  future  baseline  operational  testing  and 
reporting  with  intended  ship-deployment  schedules  to 
ensure  that  testing  and  reporting  is  completed  prior  to 
deployment. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Continue  to  develop  an  end-to-end  M&S  suite  of  the 
Aegis  Combat  System  that  may  be  used,  in  conjunction 
with  operationally  realistic  testing  conducted  on  an 
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Aegis- equipped  SDTS,  to  assess  the  PRA  requirements  of 
the  Aegis  cruisers  and  destroyers. 

2.  Provide  the  necessary  funding  to  support  the  procurement 
of  an  Advanced  Air  and  Missile  Defense  Radar  and 
Aegis-equipped  SDTS  that  is  needed  to  support  Aegis 
Modernization,  Advanced  Air  and  Missile  Defense  Radar, 
DDG  51  Flight  III,  and  Evolved  SeaSparrow  Missile 
Block  2  operational  testing. 

3.  Characterize  Aegis  Baseline  9A/C  and  soon-to-be-deployed 
NIFC-CA  FTS  Increment  I  capability  against  operationally 
realistic  ASCM  threats  as  soon  as  possible. 


4.  Continue  to  improve  Aegis  ship’s  capability  to  counter 
high-speed,  surface  threats  in  littoral  waters. 

5.  Submit,  for  DOT&E  approval,  a  Test  and  Evaluation  Master 
Plan  that  describes  and  resources  adequate  operational 
testing  of  future  NIFC-CA  FTS  increments  before  such 
capabilities  are  deployed. 

6.  For  Baseline  9A,  develop  and  deploy  necessary 
cybersecurity  corrective  actions  and  verify  correction  with  a 
follow-on  operational  cybersecurity  test. 
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AGM-88E  Advanced  Anti-Radiation  Guided  Missile 

(AARGM)  Program 


Executive  Summary 

•  The  Advanced  Anti-Radiation  Guided  Missile  (AARGM) 
remains  operationally  suitable,  but  not  operationally  effective 
due  to  multiple  deficiencies  discovered  during  IOT&E  in 
FY11-12. 

•  The  contractor  completed  test  flights  during  FY14  and 
delivered  sequential  software  versions  R1.0,  R2.0,  and  R2.1  to 
the  Navy. 

•  In  August  2014,  the  program  conducted  an  Integrated  Test 
Readiness  Review.  Based  on  the  delivery  of  software  version 
R2. 1,  AARGM  performance  during  contractor  test  flights, 
and  modeling  and  simulation  analysis,  the  AARGM  program 
manager  authorized  entry  into  integrated  testing. 

•  There  were  no  operational  test  events  scheduled  or  conducted 
during  FY14. 

System 

•  AARGM  is  the  follow-on  to  the  AGM-88B/C  High-Speed 
Anti-Radiation  Missile  (HARM)  using  a  new  guidance 
section  and  modified  HARM  control  section  and  fins.  The 
Navy  intends  to  employ  AARGM  on  F/A-18C/D/E/F  and 
EA-18G  platforms. 

•  AARGM  incorporates  digital  Anti-Radiation  Homing,  GPS, 
Millimeter  Wave  guidance,  and  a  Weapon  Impact  Assessment 
transmitter. 

•  Anti-Radiation  Homing  improvements  include  an  increased 
field-of-view  and  increased  detection  range  compared  to 
HARM. 

•  The  GPS  allows  position  accuracy  in  location  and  time;  the 
Weapons  Impact  Assessment  capability  allows  transmission  of 
real-time  hit  assessment  via  a  national  broadcast  data  system. 

•  Millimeter  Wave  radar  technology  allows  target  discrimination 
and  guidance  during  the  terminal  flight  phase. 


•  The  Navy  expects  the  AARGM  Block  1  Upgrade  to  deliver 
Full  Operational  Capability,  including  Block  0  capability 
improvements  and  software  changes  to  provide  deferred 
capability  requirements  and  address  deficiencies  identified 
during  IOT&E. 

Mission 

Commanders  employ  aircraft  equipped  with  AARGM  to  conduct 
pre-planned,  on-call,  and  time-sensitive  reactive  anti-radiation 
targeting  for  the  suppression,  degradation,  and  destruction  of 
radio  frequency-enabled  surface-to-air  missile  defense  systems. 
Commanders  receive  real-time  Weapons  Impact  Assessments 
from  AARGM  via  a  national  broadcast  data  system. 

Major  Contractor 

Alliant  Techsystems,  Defense  Electronics  Systems 
Division  -  Northridge,  California 


Activity 

•  There  were  no  operational  test  events  scheduled  or  conducted 
during  FY14. 

•  During  FY14,  the  Program  Office  and  the  Commander, 
Operational  Test  and  Evaluation  Force  (COTF),  with 
DOT&E  oversight,  developed  an  FOT&E  framework  and 
Integrated  Test  Plan  to  adequately  test  deferred  capabilities 
and  deficiencies  discovered  during  developmental  test  and 
evaluation  and  IOT&E. 

•  COTF  is  currently  reviewing  the  AARGM  Test  and  Evaluation 
Master  Plan  for  Block  1  FOT&E,  which  should  reflect  the 
agreed-upon  framework. 


•  The  contractor  conducted  flight  tests  in  FY14  and  delivered 
sequential  software  versions  R1.0,  R2.0,  and  R2. 1  to  the 
Navy.  In  August  2014,  the  AARGM  program  conducted  an 
Integrated  Test  Readiness  Review.  Based  on  the  delivery 
of  software  version  2. 1  and  AARGM  performance  during 
contractor  testing  and  modeling  and  simulation  analysis,  the 
AARGM  program  manager  authorized  AARGM  to  enter 
integrated  testing. 

•  AARGM  entered  integrated  test  in  August  2014  and  completed 
four  captive-carry  test  events.  During  analysis  of  the  missile 
performance  data,  the  Navy  determined  that  a  software  update 
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is  required  and  delayed  the  planned  captive-carry  and  live 
fire  test  events  for  the  remainder  of  1QFY15.  Integrated  test 
flights  are  expected  to  resume  in  2QFY15. 

Assessment 

•  The  FY14  status  remains  unchanged  from  the  FY13  report. 

•  AARGM  is  operationally  suitable,  but  not  operationally 
effective.  The  details  of  these  deficiencies  are  discussed  in  the 
classified  DOT&E  IOT&E  report  published  in  August  2012. 

•  The  AARGM  program  has  entered  integrated  test  based 
on  the  delivery  of  the  R2.1  software  and  contractor  and 
developmental  testing.  The  test  design  and  identified 
resources  should  provide  a  rigorous  evaluation  of  the 
corrections  of  deficiencies  discovered  in  IOT&E  and  the 


deferred  Key  Performance  Parameter.  The  early  integrated 
testing  of  captive-carry  and  live-fire  events  are  designed  to 
provide  insight  and  exposure  to  all  capabilities  and  conditions. 
These  initial  test  events  should  give  an  early  indication  of  the 
performance  of  the  missile  and  stability  of  the  system. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  satisfied  the 
previous  recommendation. 

•  FY 14  Recommendations.  The  Navy  should: 

1.  Complete  the  integrated  test  period  as  planned  for  FY15. 

2.  Complete  the  detailed  plan  for  FOT&E  based  on  the  utility 
of  integrated  test  data  for  operational  test  objectives. 
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AIM-9X  Air-to-Air  Missile  Upgrade 


Executive  Summary 

•  On  June  9,  2014,  the  Program  Executive  Officer  (PEO) 
recertified  AIM-9X  Block  II  to  resume  IOT&E.  The  Navy 
and  Air  Force  executed  18  operational  test  live  missile  shots 
(5  conducted  as  integrated  test  events),  including  2  repeated 
shots  due  to  a  previous  failure,  and  21  captive-carry  events. 

Of  the  18  live  missile  shots  attempted,  14  met  test  objectives, 
one  misfired,  one  experienced  a  known  hardware  failure,  and 
2  resulted  in  wide  misses. 

•  AIM-9X  Block  II  with  Operational  Flight  Software  (OFS) 
9.313  has  over  950  operating  hours  with  zero  recorded 
failures.  Testing  results  to  date  are  encouraging;  however, 
insufficient  data  were  available  to  provide  statistical 
confidence  in  the  system  reliability.  DOT&E  will  continue  to 
track  reliability  in  the  IOT&E. 

•  The  Navy  and  Air  Force  originally  began  AIM-9X  Block  II 
IOT&E  with  OFS  9.311  on  April  27,  2012.  On  July  29,  2013, 
the  PEO  formally  decertified  AIM-9X  Block  II  due  to 

two  major  deficiencies  discovered  and  documented  during 
IOT&E  that  affected  missile  performance.  An  extensive 
government  and  contractor  investigation  identified  hardware 
reliability  deficiencies  with  the  inertial  measurement  unit. 

The  contractor  implemented  an  improved  production  process 
and  updated  the  missile  software  (OFS  9.313)  to  address  the 
two  primary  deficiencies,  as  well  as  several  other  performance 
issues. 

System 

•  AIM-9X  is  the  latest  generation  short-range,  heat-seeking, 
air-to-air  missile.  The  currently  fielded  version  of  the  missile 
is  AIM-9X  Block  I,  OFS  8.220,  which  includes  limited 
lock- on-after-launch,  full  envelope  off  boresight  capability 
without  a  helmet-mounted  cueing  system,  and  improved  flare 
rejection  performance. 

•  AIM-9X  is  highly  maneuverable,  day/night  capable,  and 
includes  the  warhead,  fuze,  and  rocket  motor  from  the 
previous  AIM-9M  missile. 

•  AIM-9X  added  a  new  imaging  infrared  seeker,  vector 
controlled  thrust,  digital  processor,  and  autopilot. 

•  F-15C/D,  F-16C/D,  and  F/A-18C/D/E/F  aircraft  are  capable  of 
employing  the  AIM-9X. 


•  The  AIM-9X  Block  II  is  the  combination  of  AIM-9X-2 
hardware  and  OFS  9.3. 

-  AIM-9X-2  is  the  latest  hardware  version  and  is  designed 
to  prevent  parts  obsolescence  and  provide  processing 
capability  for  the  OFS  9.3  upgrade.  The  AIM  9X-2  missile 
includes  a  new  processor,  a  new  battery,  an  electronic 
ignition  safe/arm  device,  and  the  DSU-41/B  Active  Optical 
Target  Detector  fuze/datalink  assembly. 

-  OFS  9.3  is  a  software  upgrade  that  is  intended  to  add 
trajectory  management  to  improve  range,  datalink  with  the 
launching  aircraft,  improved  lock-on-after-launch,  target 
re-acquisition,  and  improved  fuzing. 

Mission 

Air  combat  units  use  the  AIM-9X  to: 

•  Conduct  short-range  offensive  and  defensive  air-to-air  combat 

•  Engage  multiple  enemy  aircraft  types  with  passive  infrared 
guidance  in  the  missile  seeker 

•  Seek  and  attack  enemy  aircraft  at  large  angles  away  from  the 
heading  of  the  launch  aircraft 

Major  Contractor 

Raytheon  Missile  Systems  -  Tucson,  Arizona 


Activity 

•  At  the  beginning  of  FY14,  the  Navy  and  Raytheon  Missile 
Systems  were  continuing  investigations  into  AIM-9X 
Block  II  deficiencies  found  during  IOT&E  that  had  resulted 
in  PEO  decertification  of  the  program  from  testing  on 
July  29,  2013.  Between  the  start  of  IOT&E  on  April  27,  2012, 


and  decertification,  the  Navy  originally  completed  18  of 
22  planned  captive  carry  events,  5  of  9  planned  live  missile 
shots,  and  1  repeat  test  shot.  The  Air  Force  originally 
completed  18  of  22  captive-carry  events  and  6  of  8  live  missile 
shots.  Of  the  12  live  missile  shots  conducted  during  IOT&E 
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prior  to  decertification,  7  terminated  within  lethal  radius  of  the 
target. 

•  The  Program  Office  and  Raytheon  Missile  Systems  completed 
the  investigation  and  implemented  hardware  and  software 
solutions  to  address  the  two  primary  deficiencies.  From 
March  20  through  May  13,  2014,  the  Navy  and  Air  Force 
executed  5  successful  integrated  test  live  missile  shots  and 

13  captive- carry  missions  to  test  missile  performance  with 
changes  to  the  inertial  measurement  unit  hardware  and  the 
OFS. 

•  On  June  5,  2014,  the  Navy  completed  an  Operational  Test 
Readiness  Review  and  the  PEO  certified  AIM-9X  Block  II 
with  OFS  9.3 13  for  IOT&E.  DOT&E  approved  a  test  plan 
change  reducing  the  number  of  captive-carry  missions  to 
28  (14  per  Service)  and  removed  one  of  the  17  live  missile 
tests  from  the  originally  approved  IOT&E  plan. 

•  The  Navy  completed  9  of  14  planned  captive-carry  events  and 
9  of  9  live  missile  shots,  plus  2  repeated  shots.  The  Air  Force 
completed  12  of  14  planned  captive-carry  events  and  7  of 

7  live  missile  shots.  Of  the  18  live  missile  shots,  14  terminated 
within  lethal  radius  of  the  target,  2  resulted  in  wide  misses, 

1  experienced  a  known  hardware  failure,  and  1  misfired. 

These  results  include  the  five  integrated  test  live  missile  shots, 
which  terminated  within  lethal  radius. 

•  The  Program  Office  conducted  the  IOT&E  in  accordance  with 
the  DOT&E-approved  Test  and  Evaluation  Master  Plan. 

Assessment 

•  Prior  to  decertification  from  testing,  7  of  12  operational  test 
AIM-9X  Block  II  (OFS  9.311)  shots  guided  to  within  lethal 
radius  of  the  drone.  The  developmental  testing  record  was 

9  of  12  shots  within  lethal  radius;  however,  1  missile  did  not 
receive  a  fuze  pidse. 


•  The  aircrew  reported  that  AIM-9X  Block  II  Helmet-less  High 
Off-Boresight  performance  with  OFS  9.313  is  on  par  with 
AIM-9X  Block  I  performance.  The  Capability  Production 
Document  requires  Block  II  performance  be  equal  to  or  better 
than  baseline  AIM-9X  performance. 

•  Based  upon  live  missile  testing  performance  with  enhanced 
inertial  measurement  unit  production  processes  and 

OFS  9.313,  the  Navy  and  Raytheon  have  resolved  the 
previously  identified  fly-out  deficiency  that  significantly 
affected  Probability  of  Kill. 

•  The  results  of  AIM-9X  Block  II  testing  with  OFS  9.3 13  to  date 
are  encouraging;  however,  insufficient  data  were  available  to 
provide  statistical  confidence  in  system  reliability.  The  Navy 
and  Air  Force  accomplished  over  950  operating  hours  with 
zero  recorded  failures.  When  the  PEO  decertified  AIM-9X 
Block  II  from  IOT&E,  the  Navy  and  Air  Force  had  conducted 
6,353  total  operating  hours  with  22  failures,  resulting  in  a 
Mean  Time  Between  Critical  Failure  of  288.79  hours.  This 
was  well  below  the  reliability  growth  curve  to  achieve  the 
requirement  of  500  hours  Mean  Time  Between  Critical  Failure 
at  80,000  hours.  DOT&E  will  continue  to  track  reliability  in 
the  IOT&E. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  addressed 
the  previous  recommendations. 

•  FY 14  Recommendation. 

1.  The  Navy  should  work  closely  with  DOT&E  and  the 
Service  Operational  Test  Agencies  to  establish  the  plan, 
requirements,  and  resources  for  OFS  9.400  testing, 
including  the  associated  Test  and  Evaluation  Master  Plan 
update. 
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AN/SQQ-89A(V)15  Integrated  Undersea  Warfare  (USW) 

Combat  System  Suite 


Executive  Summary 

•  Operational  testing  of  the  Advanced  Capability  Build  2011 
(ACB-11)  variant  began  in  FY14  and  is  expected  to  conclude 
in  FY15.  However,  the  Navy  has  not  yet  scheduled  all 
required  IOT&E  events.  The  Navy  completed  limited,  at-sea 
testing  in  FY14  in  conjunction  with  two  fleet-training  events. 

•  In  December  2014,  DOT&E  issued  a  classified  Early  Fielding 
Report  on  the  ACB-11  variant  of  AN/SQQ-89A(V)15 
Integrated  Undersea  Warfare  Combat  System  Suite.  The 
report  was  submitted  due  to  the  installation  of  the  ACB-11 
variant  on  ships  that  deployed  prior  to  IOT&E.  From  the  data 
collected,  DOT&E  concluded  the  system  demonstrated  some 
capability  to  detect  submarines  and  incoming  U.S.  torpedoes 
in  deep  water.  However,  no  data  were  available  to  assess  its 
capability  in  shallow  water,  an  area  of  significant  interest  due 
to  the  prevalence  of  submarines  operating  in  littoral  regions. 
Also,  no  data  were  available  to  assess  performance  against 
threat  torpedoes. 

System 

•  AN/SQQ-89A(V)15  is  the  primary  Undersea  Warfare  system 
used  aboard  U.S.  Navy  surface  combatants  to  locate  and 
engage  threat  submarines.  AN/SQQ-89A(V)15  is  an  open 
architecture  system  that  includes  biannual  software  upgrades 
(ACBs)  and  four-year  hardware  upgrades  called  Technology 
Insertions. 

•  AN/SQQ-89A(V)15  uses  active  and  passive  sonar  to  conduct 
Anti-Submarine  Warfare  (ASW)  search.  Received  acoustic 
energy  is  processed  and  displayed  to  support  operator 
detection,  classification,  localization,  and  tracking  of  threat 
submarines. 

•  AN/SQQ-89A(V)15  uses  passive  sonar  (including  acoustic 
intercept)  to  provide  early  warning  of  threat  torpedoes. 

•  The  Navy  intends  for  the  program  to  provide  improvement  in 
sensor  display  integration  and  automation,  reduction  in  false 
alerts,  and  improvement  in  onboard  training  capability  to 
better  support  operation  within  littoral  regions  against  multiple 
sub-surface  threats. 

•  The  system  consists  of: 

-  Acoustic  sensors  -  hull-mounted  array,  multi-function 
towed  array  (TB-37)  including  a  towed  acoustic  intercept 
array,  calibrated  reference  hydrophone,  helicopter,  and/or 
ship  deployed  sonobuoys 


-  Functional  segments  used  for  processing  and  display  of 
active,  passive,  and  environmental  data 

-  Interface  to  Aegis  Combat  System  for  Mk  46  and 

Mk  54  torpedo  prosecution  using  surface  vessel  torpedo 
tubes,  Vertical  Launch  Anti-Submarine  Rocket,  or 
SH-  60B/MH-  60R  helicopters 

•  The  system  is  deployed  on  a  DDG  51  class  destroyer  or  CG  47 
class  cruiser  host  platform. 

Mission 

•  Maritime  Component  Commanders  employ  surface 
combatants  with  AN/SQQ-89A(V)15  as  escorts  to  high-value 
units  to  protect  against  threat  submarines  during  transit. 

•  Maritime  Component  Commanders  use  surface  combatants 
with  AN/SQQ-89A(V)15  to  conduct  area  clearance  and 
defense,  barrier  operations,  and  ASW  support  during 
amphibious  assault. 

•  Theater  Commanders  use  surface  combatants  with 
AN/SQQ-89A(V)15  to  support  theater  ASW  prosecution  of 
threat  submarines. 

•  Unit  Commanders  use  AN/SQQ-89A(V)15  to  support 
self- protection  against  incoming  threat  torpedoes. 

Major  Contractor 

Lockheed  Martin  Mission  Systems  and  Training  -  Manassas, 

Virginia 
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Activity 

•  In  January  2013,  DOT&E  sent  a  memorandum  to  the 
Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  outlining  the  need  for  a  threat  torpedo  surrogate 
to  support  operational  testing  of  the  AN/SQQ-89A(V)15.  In 
October  2013,  the  Navy  commenced  a  fomial  study  to  identify 
gaps  in  currently  available  torpedo  surrogates’  capability  to 
represent  threat  torpedoes.  The  study  was  also  intended  to 
provide  an  analysis  of  alternatives  for  either  the  modification 
of  current  surrogates  or  the  development  of  new  surrogates  to 
overcome  these  identified  gaps. 

•  In  August  2014,  DOT&E  approved  the  Test  and  Evaluation 
Master  Plan  covering  the  ACB- 11  variant.  ACB-11 
operational  testing  will  include  at-sea  evaluations  focusing 
on  ASW  and  torpedo  detection,  particularly  in  shallow-water 
(generally  defined  as  water  that  is  less  than  100  fathoms  in 
depth),  littoral  environments  that  have  not  been  evaluated 

in  prior  variants.  Testing  will  also  include  a  cybersecurity 
evaluation. 

•  In  December  2014,  DOT&E  issued  a  classified  Early  Fielding 
Report  for  the  ACB-11  variant  of  AN/SQQ-89A(V)15 
Integrated  Undersea  Warfare  Combat  System  Suite.  The 
report  was  issued  due  to  the  installation  of  the  ACB-11  variant 
on  ships  that  deployed  prior  to  IOT&E. 

•  The  Commander,  Operational  Test  and  Evaluation 
Force  commenced  IOT&E  on  the  ACB-11  variant  in 
May  2014.  Testing  was  conducted  in  accordance  with  a 
DOT&E- approved  test  plan  and  included  ASW  transit 
search  and  area  search  operations  using  AN/SQQ-89A(V)  15 
onboard  a  DDG  5 1  class  destroyer.  Testing  was  conducted  in 
conjunction  with  the  following  two  fleet  events: 

-  Submarine  Command  Course  40  Anti-Surface  Warfare 
events  at  the  Navy’s  Atlantic  Undersea  Test  and  Evaluation 
Center. 

-  Submarine  Command  Course  40  ASW  events  at  the 

Fort  Pierce,  Florida,  Operations  Area.  This  testing  focused 
on  torpedo  employment  in  shallow  water. 

•  The  Navy  has  not  yet  scheduled  the  dedicated  IOT&E. 

Assessment 

•  The  final  assessment  of  ACB-11  is  not  complete,  as  testing 
is  expected  to  continue  through  FY15.  DOT&E ’s  classified 
Early  Fielding  Report  concluded  the  following  regarding 
performance: 

-  The  ACB- 1 1  variant  appears  to  be  meeting  program 
performance  metrics  for  submarine  detection  and 
classification  in  deep-water  environments.  This 
assessment  is  made  with  low  confidence  due  to  limited 
data  collection.  Also,  the  data  were  insufficient  to 
determine  if  this  detection  capability  and  accompanying 
operator  classifications  would  translate  to  an  effective 
prosecution  of  the  threat  submarine. 

-  The  ACB- 1 1  variant  demonstrated  some  capability  to 
detect  U.S.  torpedoes  at  a  program-defined  range  that 
is  intended  to  support  a  meaningful  torpedo  evasion. 


However,  the  data  were  insufficient  to  determine  if  the 
observed  torpedo  detection  ranges  will  support  effective 
torpedo  evasion.  No  data  were  available  to  assess 
performance  against  quieter,  modern  torpedoes. 

-  The  ACB-11  variant  is  currently  not  suitable  due  to  low 
operational  availability.  Extensive  logistic  delays  limited 
system  capability  throughout  the  majority  of  the  time  frame 
evaluated.  A  primary  contributor  was  a  significant  delay 

in  the  repair  of  the  Multi-Function  Towed  Array  (MFTA) 
because  of  a  limited  inventory  of  spare  arrays;  array  repair 
is  primarily  achieved  through  replacement  of  the  MFTA  in 
port. 

-  Insufficient  data  were  collected  to  confidently  determine 
performance  for  the  ACB-11  variant  against  real-world, 
diesel  submarines  and  nuclear  submarines.  Further,  no 
assessment  can  be  made  against  the  smaller  midget  and 
coastal  diesel  submarines  due  to  the  Navy  having  no  test 
surrogates  to  represent  this  prevalent  threat. 

•  Analysis  of  the  few  completed  IOT&E  events  is  ongoing. 
Preliminary  analysis  indicates  that  the  ACB-11  variant  has 
some  capability  to  detect  submarines  in  shallow  water. 
However,  the  fleet  exercise  did  not  support  the  necessary 
ranges  to  assess  detection  against  system  requirements. 

•  The  ability  of  surface  combatants  employing  the  ACB- 1 1 
variant  to  avoid  torpedoes  can  only  partially  be  assessed  due 
to  significant  differences  in  U.S.  torpedoes  and  torpedoes 
employed  by  other  nations. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  is 
making  progress  and  should  continue  to  address  FY13 
recommendations.  The  Navy  has  started  the  process  for 
identifying  existing  gaps  between  threat  torpedoes  and 
available  torpedo  surrogates  in  operational  testing.  However, 
the  Navy  still  needs  to: 

1 .  Complete  an  analysis  and  develop  a  plan  to  overcome 
gaps  between  threat  torpedoes  and  torpedoes  available  for 
operational  testing. 

2.  Schedule  and  complete  dedicated  IOT&E  test  events  in 
shallow  water. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Develop  and  integrate  high-fidelity  trainers  and  realistic, 
in-water  test  articles  to  improve  training  and  proficiency  of 
operators  in  ASW  search  and  track  of  threat  submarines, 
including  midget  and  coastal  diesel  submarines. 

2.  Include  and  assess  capability  of  AN/SQQ-89A(V)15  during 
all  opportunities  against  real-world  diesel  submarines  to 
determine  performance  differences  from  that  observed 
against  U.S.  nuclear  submarines. 

3.  Pre -position  spare  TB-37  MFTA  and  spare  MFTA  modules 
at  appropriate  forward-operating  ports  to  minimize  logistic 
delays. 

4.  Address  the  four  additional  classified  recommendations 
listed  in  the  December  2014  Early  Fielding  Report. 
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Armored  Tactical  Vehicles  -  Naval 


Executive  Summary 

•  The  Marine  Corps  has  developed  a  special  purpose  kit 
to  improve  protection  from  under-vehicle  attacks  for  the 
Logistics  Vehicle  System  Replacement  (LVSR)  truck.  For 
the  wrecker  variant,  the  Underbody  Improvement  Kit  (UIK) 
has  several  unique  design  features  to  accommodate  the 
self- recovery  winch. 

•  The  Marine  Corps  completed  two  LVSR  system-level 
underbody  blast  tests  in  June  2014  at  Aberdeen  Test  Center, 
Maryland;  the  data  indicate  that  the  UIK  increases  crew 
protection. 

•  The  ballistic  test  phase  addressed  the  crew/occupant 
vulnerabilities  of  the  Medium  Tactical  Vehicle  Replacement 
(MTVR)  Engineering  Change  Proposal  (ECP)  vehicles  against 
specific  underbody  threats.  These  events  were  selected  for  the 
MTVR  ECP  vehicles  based  on  the  previous  live  fire  testing 

of  the  MTVR,  and  designed  to  provide  comparative  data  for 
evaluation  of  seating  performance  and  crew  force  protection. 

•  The  Marine  Corps  completed  two  of  the  three  planned  MTVR 
ECP  test  events  in  March  2014  at  Aberdeen  Test  Center, 
Maryland,  before  the  test  series  was  stopped  for  assessment 
and  redesign. 

System 

•  The  Marine  Corps  Armored  Tactical  Vehicle  Programs  include 
the  LVSR  and  the  MTVR  trucks. 

•  The  LVSR  is  a  family  of  heavy  trucks,  including  the  LVSR 
Cargo  and  Tractor  platforms  and  the  LVSR  Wrecker  variant. 
LVSRs  are  capable  of  transporting  18  tons  off-road  and 
22.5  tons  on-road.  The  LVSR  Wrecker  has  several  unique 
design  characteristics  to  accommodate  the  self-recovery 
winch.  To  improve  the  vehicles’  survivability  against 
underbody  blast  threats,  a  UIK  is  being  designed  by  the 
Marine  Corps,  though  not  currently  planned  for  procurement. 

•  The  MTVR  is  a  family  of  medium  trucks,  equipped  with 
armor  protection  kits,  which  are  capable  of  transporting 

6  tons  off-road  and  12.2  tons  on-road.  Other  ECPs  include 
energy- absorbing  seats  and  floor  mats,  emergency  egress 
windshields,  and  an  automatic  fire  extinguishing  system. 

Mission 

The  Marine  Corps  employs  truck  systems  as  multi-purpose 

transportation  and  unit  mobility  vehicles  in  combat,  combat 

support,  and  combat  service  support  units. 


LVSR  -  Logistics  Vehicle  System  Replacement 


MTVR  -  Medium  Tactical  Vehicle  Replacement 


•  The  LVSR  is  the  heavy  lift  transport  capability  within  all 
elements  of  the  Marine  Air  Ground  Task  Force,  which 
includes  transporting  bulk  fuel  and  water,  ammunition,  cargo, 
tactical  bridging,  containers,  combat  engineer  vehicles,  and 
heavy  wrecker  capability. 

•  The  MTVR  is  the  prime  mover  for  the  howitzer,  fuel  and 
water  assets,  troops,  and  a  wide  variety  of  equipment. 

Major  Contractor 

LVSR  and  MTVR:  Oshkosh  Defense  -  Oshkosh,  Wisconsin 


Activity 

•  Developmental  and  system-level  testing  of  the  LVSR  Wrecker 
UIK  integration  concluded  in  3QFY14.  These  tests  included 


two  underbody  blast  events  conducted  at  Aberdeen  Test 
Center,  Maryland.  The  LVSR  Program  Office  provided 
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two  surrogate  wrecker  assets  to  test  and  characterize  the 
force  protection  capabilities  and  vehicle  vulnerability  against 
underbody  blast  threats. 

•  The  Program  Manager  Medium  and  Heavy  Tactical  Vehicles 
(PM  M&HTV)  started  an  effort  to  evaluate  ECP  components 
for  the  MTVR.  This  test  phase  is  designed  to  demonstrate 
blast  performance  of  candidate  MTVR  energy-absorbing  seats 
and  floor  mats,  and  verify  blast-induced  shock  resistance  and 
safety  of  other  integrated  changes,  including: 

-  Command,  Control,  Communications,  Computers,  and 
Intelligence  equipment 

-  The  selected  emergency  egress  windshields 

-  An  automatic  fire  extinguishing  system 

•  The  Marine  Corps  conducted  two  of  the  three  planned  MTVR 
underbody  blast  test  events  at  Aberdeen  Test  Center,  Maryland, 
in  2QFY14. 

Assessment 

•  Testing  and  analysis  confirm  that  the  LVSR  Wrecker  UIK 
increases  crew  protection  against  some  under-vehicle  mine 
strikes.  The  program  manager  has  no  plans  to  procure  and 
field  the  LVSR  UIK. 

•  Designs  of  the  UIK  have  been  refined  and  qualified  through 
the  recent  LVSR  UIK  testing,  and  the  Cargo,  Tractor,  and 


Wrecker  vehicles  with  UIKs  added  have  shown  improved  crew 
survivability  against  underbody  blast  events. 

•  Results  from  the  initial  MTVR  ECP  testing  in  2QFY14 
indicated  no  noticeable  improvement  in  crew  survivability 
and  therefore  testing  was  stopped.  PM  M&HTV  is  currently 
considering  a  redesign  of  the  legacy  survivability  kit  and  ECP 
(seat  and  floor  mat)  components  and  will  notify  DOT&E  if  it 
plans  to  proceed  with  further  changes  to  the  vehicle. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Marine  Corps 
addressed  the  previous  recommendations  by  conducting  live 
fire  testing  of  armor  upgrades  and  design  changes. 

•  FY 14  Recommendations. 

1 .  As  the  Marine  Corps  has  decided  to  delay  procurement  of 
the  LVSR  UIKs,  future  missions  requiring  LVSR  vehicles 
with  UIK  should  evaluate  expected  threats  for  changes  prior 
to  fielding. 

2.  PM  M&HTV  should  reevaluate  MTVR  energy-absorbing 
seat  upgrade  options  and  plan  for  additional  tests  of  these 
options  when  ready. 
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Cobra  King 

(formerly  Cobra  Judy  Replacement) 


Executive  Summary 

•  The  Air  Force  Operational  Test  and  Evaluation  Center  and 
the  Navy,  Commander  Operational  Test  and  Evaluation  Force 
conducted  a  Multi-Service  Operational  Test  and  Evaluation 
(MOT&E)  of  Cobra  King  from  September  2013  through 
April  2014. 

•  Testing  included  data  collection  activities  on  balloon-borne 
calibration  spheres,  satellites,  and  two  domestic 
intercontinental  ballistic  missile  launches. 

•  On  March  31,2014,  the  Air  F orce  accepted  Cobra  King  as  an 
Initial  Operational  Capability. 

•  In  July  2014,  DOT&E  published  a  classified  MOT&E  report 
that  assessed  Cobra  King  as  operationally  effective  and 
operationally  suitable,  matching  or  exceeding  the  legacy 
Cobra  Judy  performance. 

System 

•  Cobra  King  is  a  mobile  radar  suite  installed  on  the 
USNS  Howard  O.  Lorenzen. 

•  Cobra  King  replaces  the  original  Cobra  Judy  system,  which 
has  been  deployed  since  1981  and  has  reached  the  end  of  its 
service  life. 

•  The  Cobra  King  radar  suite  consists  of  steerable  S-  and 
X-band  phased  arrays,  which  expand  the  data  collection 
capability  over  the  original  system.  The  S-band  radar 
primarily  conducts  large-volume  searches  and  is  capable  of 
performing  radar  tracks  and  collections  on  a  large  number 
of  radar  targets.  The  X-band  radar  provides  high-resolution 
data  on  specific  radar  objects  of  interest  and  also  has  a  search 
capability. 

•  The  ship’s  crew  consists  of  civilian  or  contracted  Military 
Sealift  Command  personnel  responsible  for  the  navigation. 


operations,  and  maintenance  of  the  ship;  a  small,  specialized 
group  of  contractors  are  utilized  for  radar  operations.  An 
Air  Force  officer  serves  as  the  mission  commander. 

Mission 

The  DOD  uses  Cobra  King  to  conduct  treaty  monitoring  and 
verification  activities.  Additionally,  Cobra  King  can  be  used  to 
provide  data  for  comparison  with  other  sources  during  domestic 
ballistic  missile  tests. 

Major  Contractors 

•  Raytheon  Integrated  Defense  Systems  -  Sudbury,  Massachusetts 

•  Northrop  Grumman  Electronic  Systems  -  Baltimore,  Maryland 

•  VT  Flalter  Marine  -  Pascagoula,  Mississippi 


Activity 

•  From  September  2013  through  April  2014,  the  U.S.  Air  Force 
Operational  Test  and  Evaluation  Center  and  the  Commander, 
Operational  Test  and  Evaluation  Force  conducted  an  MOT&E 
for  Cobra  King. 

•  Testing  included  data  collection  activities  on  balloon-borne 
calibration  spheres,  satellites,  and  two  domestic 
intercontinental  ballistic  missile  launches.  Testing  was 
executed  in  accordance  with  a  DOT&E-approved  Test  and 
Evaluation  Master  Plan  and  test  plan. 

•  On  March  31,  2014,  the  Air  Force  accepted  Cobra  King  as  an 
Initial  Operational  Capability. 


•  In  July  2014,  DOT&E  published  a  classified  MOT&E 
report  that  assessed  Cobra  King  test  adequacy,  operational 
effectiveness,  and  suitability. 

Assessment 

•  The  Cobra  King  test  program  achieved  a  successful  balance 
between  the  use  of  modeling  and  simulation  and  the 
observation  of  balloon-borne  spheres,  satellites,  and  live 
ballistic  missile  targets.  Testing  was  adequate  to  support 
an  evaluation  of  operational  effectiveness  and  operational 
suitability. 
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•  Cobra  King  is  operationally  effective  to  accomplish  its 
mission,  matching  or  exceeding  the  legacy  Cobra  Judy 
performance  in  relevant  mission  areas  (i.e.,  preparation  and 
transit,  loiter,  data  collection,  data  analysis  and  exchange, 
storage  and  archiving,  and  cybersecurity)  during  MOT&E. 

•  Cobra  King  is  operationally  suitable  to  accomplish  its 
mission.  Based  upon  test  performance  during  MOT&E, 
Cobra  King  is  expected  to  match  the  legacy  Cobra  Judy 
system  in  reliability,  availability,  and  interoperability. 


Recommendations 

•  Status  of  Previous  Recommendations.  There  are  no  previous 
recommendations  from  the  FY13  Cobra  Judy  Replacement 
Annual  Report. 

•  FY 14  Recommendation. 

1 .  The  program  should  continue  to  address  the  deficiencies 
identified  during  MOT&E  -  and  outlined  in  DOT&E’s 
classified  report  -  to  further  improve  the  system’s 
operational  effectiveness  and  suitability. 
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Common  Aviation  Command  and  Control  System 

(CAC2S) 


Executive  Summary 

•  In  December  2013,  the  Test  and  Evaluation  Master  Plan 
(TEMP)  for  Common  Aviation  Command  and  Control  System 
(CAC2S)  Increment  I  was  updated  by  the  CAC2S  Program 
Office  to  support  the  planned  Milestone  C  review,  currently 
scheduled  for  2QFY 15. 

•  In  October  2014,  the  Marine  Corps  Operational  Test  and 
Evaluation  Activity  conducted  an  operational  assessment  (OA) 
of  CAC2S  Increment  I,  Phase  2  during  the  Weapons  and  Tactics 
Instructors’  (WTI)  exercise  at  Marine  Corps  Air  Station  Yuma, 
Arizona,  to  assess  its  ability  to  support  the  Direct  Air  Support 
Center  (DASC),  Tactical  Air  Operations  Center  (TAOC),  and 
Tactical  Air  Command  Center  (TACC)  missions.  The  OA 

was  conducted  in  accordance  with  a  DOT&E-approved  test 
plan,  and  the  results  will  be  used  to  support  a  Low-Rate  Initial 
Production  (LRIP)  decision  during  the  Milestone  C  review. 

•  During  the  OA,  CAC2S,  Increment  I,  Phase  2  demonstrated  the 
ability  to  support  mission  accomplishment  of  the  three  Marine 
Corps  aviation  C2  missions.  CAC2S  Increment  I,  Phase  2  also 
demonstrated  the  ability  to  provide  data  fusion  of  real  time, 
near-real  time,  and  non-real  time  information  onto  a  single 
tactical  display. 

System 

•  CAC2S  consists  of  tactical  shelters,  software,  and  common 
hardware.  The  hardware  components  are  expeditionary, 
common,  modular,  and  scalable,  and  may  be  freestanding, 
mounted  in  transit  cases,  or  rack-mounted  in  shelters  and/or 
general-purpose  tents  that  are  transported  by  organic  tactical 
mobility  assets. 

•  CAC2S  Increment  I  is  being  delivered  in  two  phases.  Phase  I 
previously  delivered  hardware  and  software  to  fully  support 
the  DASC  mission  requirements  and  partially  support  TAOC 
mission  requirements.  Phase  2  combines  the  three  legacy 
Phase  1  systems  into  two  functional  subsystems  and  fully 
supports  the  requirements  of  the  DASC,  TACC,  and  TAOC. 

-  The  Communication  Subsystem  (CS)  provides  the 
capability  to  interface  with  internal  and  external 
communication  assets  and  the  means  to  control  their 
operation. 

-  The  Aviation  Command  and  Control  System  (AC2S) 
provides: 

■  The  operational  command  post  and  functionality  to 
support  mission  planning,  decision  making,  and  execution 
tools  in  support  of  all  functions  of  Marine  Aviation 

■  An  open  architecture  interface  capable  of  integrating 
emerging  active  and  passive  sensor  technology  for  organic 
and  non-organic  sensors  to  the  Marine  Air  Command  and 
Control  System  (MACCS) 


HF  -  High  Frequency 

JREAP  -  Joint  Range  Extension  Application  Protocol 

MIDS  ■  Multi-Functional  Information  Distribution  System 

OPFAC  -  Operations  Facility 

SATCOM  -  Satellite  Communications 

TDL  -  Tactical  Data  Link 

UHF  -  Ultra  High  Frequency 


■  The  capability  for  the  display  of  real  time,  near-real  time, 
and  non-real  time  sensor  data  to  support  C2  of  Marine 
Air-Ground  Task  Force  (MAGTF)  aviation  assets 


Mission 

•  The  MAGTF  Commander  will  employ  Marine  Corps  aviation 
C2  assets  to  include  the  DASC,  the  TAOC,  and  the  TACC 
equipped  with  CAC2S  to  integrate  Marine  Corps  aviation 
into  joint  and  combined  air/ground  operations  in  support  of 
Operational  Maneuver  from  the  Sea,  Sustained  Operations 
Ashore,  and  other  expeditionary  operations. 

•  The  MAGTF  Commander  will  execute  C2  of  assigned  assets 
afloat  and  ashore  in  a  joint,  allied,  or  coalition  operational 
environment  by  using  CAC2S  capabilities  to: 

-  Share  mission-critical  voice,  video,  sensor,  and  C2  data 
and  information  in  order  to  integrate  aviation  and  ground 
combat  planning  and  operations 

-  Display  a  common,  real,  and  near-real  time  integrated 
tactical  picture  with  the  timeliness  and  accuracy  necessary 
to  facilitate  the  control  of  friendly  assets  and  the 
engagement  of  threat  aircraft  and  missiles 

-  Provide  fusion  of  real  time,  near-real  time,  and  non-real 
time  information  in  support  of  the  MAGTF 

-  Access  theater  and  national  intelligence  sources  from  a 
multi-function  C2  node 

-  Standardize  Air  Tasking  Order  (ATO)  and  Airspace  Control 
Order  generation,  parsing,  interchange,  and  dissemination 
throughout  the  MAGTF  and  theater  forces  by  using  the 
joint  standard  for  ATO  interoperability 


CAC2S 
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Major  Contractors 

•  Phase  1  *  Phase  2 

-  Government  Integrator:  -  Prime  Contractor  (no  Government  Integrator): 

Naval  Surface  Warfare  Center  -  Crane,  Indiana  General  Dynamics  -  Scottsdale,  Arizona 

-  Component  Contractor: 

Raytheon-Solipsys  -  Fulton,  Maryland 

-  Component  Contractor: 

General  Dynamics  -  Scottsdale,  Arizona 


Activity 

•  The  Marine  Corps  awarded  the  Engineering,  Manufacturing 
and  Development  contract  for  CAC2S  Phase  2  to  General 
Dynamics  C4  Systems. 

•  The  CAC2S  Program  Office  held  a  Critical  Design  Review  for 
Phase  2  during  1QFY14. 

•  The  CAC2S  Program  Office  completed  Developmental  Test 
(DT)  B-l,  DT  B-2,  and  DT  B-3  between  1Q-4QFY14. 

•  In  December  2013,  the  CAC2S  Program  Office  updated 
the  TEMP  for  CAC2S  Increment  I  to  support  the  planned 
Milestone  C  review.  DOT&E  approved  the  TEMP  in 
January  2014. 

•  In  October  2014,  the  Marine  Corps  Operational  Test  and 
Evaluation  Activity  conducted  an  OA  of  CAC2S  Increment  I, 
Phase  2  during  the  WTI  exercise  at  Marine  Corps  Air  Station 
Yuma,  Arizona.  The  OA  evaluated  the  ability  of  CAC2S 
Increment  I,  Phase  2  to  support  the  DASC,  TAOC,  and 
TACC  missions.  The  OA  was  conducted  in  accordance  with 
a  DOT&E-approved  test  plan,  and  the  results  will  be  used  to 
support  a  LRIP  decision  during  the  Milestone  C  review. 

•  Milestone  C  for  CAC2S  Phase  2  is  scheduled  for  2QFY15. 

Assessment 

•  Based  on  qualitative  evaluation  during  the  OA,  CAC2S 
Increment  I,  Phase  2  successfully  demonstrated  the  ability  to 
support  the  primary  mission  areas  for  all  three  agencies:  direct 
air  support  for  the  DASC,  control  of  aircraft  and  missiles  for 
the  TAOC,  and  providing  C2  aviation  and  planning  support  for 
the  MAGTF  commander  in  the  TACC. 

•  Increment  I,  Phase  2  support  of  the  TACC  was  limited 
primarily  to  the  current  operations  cell  due  to  the  scope  of  the 
exercise  scenario. 

•  During  the  OA,  CAC2S  Increment  I,  Phase  2  also 
demonstrated  an  ability  to  fuse  real  time,  near-real  time,  and 
non-real  time  data  onto  a  single  tactical  display,  at  medium 
operational  tempo  densities  of  aircraft  and  targets  against 
older/current  generation  threats.  However,  because  system 
interfaces  are  still  in  development,  CAC2S  connectivity  to 
the  Marine  Corps’  Composite  Tracking  Network  was  not 
demonstrated. 


•  Reliability,  availability,  and  maintainability  data  were 
collected  during  the  OA  and  the  data  indicate  that  CAC2S 
is  making  progress  toward  meeting  its  reliability  growth 
objectives. 

•  The  OA  did  not  assess  interoperability/integration  of  CAC2S 
with  the  Ground/Air  Task  Oriented  Radar  as  that  system  is  still 
undergoing  development.  However,  the  OA  did  demonstrate 
the  ability  to  connect  the  AN/TPS-59  radar  sensor  directly  to 
CAC2S  displaying  both  radar  plot  and  track  data. 

Recommendations 

•  Status  of  Previous  Recommendations.  DOT&E  previously 
recommended  that  a  balanced  use  of  air  and  ground  combat 
forces  be  used  during  future  test  venues  to  provide  a  better 
assessment  of  CAC2S  support  to  the  MAGTF  and  that 
24-hour  operations  be  conducted  to  ensure  adequate  hours  for 
assessment  of  system  reliability.  Additionally,  DOT&E  also 
recommended  that  a  Service-level  preventative  maintenance 
plan  be  developed  prior  to  conducting  the  Increment  I,  Phase  2 
IOT&E.  The  Marine  Corps  is  actively  addressing  these 
recommendations.  All  other  previous  recommendations  have 
been  addressed. 

•  FY 14  Recommendations.  The  Marine  Corps  should: 

1.  Ensure  that  IOT&E  of  CAC2S’  data  fusion  capability 
include  the  most  current  and  the  likely  projected  air  threat 
available  and  that  the  scenario  used  is  operationally  realistic 
and  of  sufficient  operational  tempo  to  adequately  stress  the 
system. 

2.  Conduct  interoperability  and  integration  testing  with 
the  Composite  Tracldng  Network  and  Ground/Air  Task 
Oriented  Radar  in  an  operationally  realistic  environment  if 
these  systems  are  sufficiently  mature  to  reduce  IOT&E  risk. 

3.  Conduct  a  Field  User  Evaluation  as  IOT&E  risk  reduction 
that  includes  all  di visions/ sections  within  the  TACC  in 
order  to  adequately  stress  the  system  and  ensure  that 
CAC2S  capabilities  meet  the  user  mission  planning 
requirements. 
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CVN-78  Gerald  R.  Ford  Class  Nuclear  Aircraft  Carrier 


Executive  Summary 

•  DOT&E’s  assessment  of  CVN-78  remains  consistent  with  the 
report  DOT&E  issued  in  December  2013,  which  was  based 
on  data  obtained  during  a  DOT&E-approved  Commander, 
Operational  Test  and  Evaluation  Force  (COTF)  operational 
assessment  completed  in  December  2013. 

•  The  Navy  submitted  the  LFT&E  Management  Plan, 

Revision  B  in  July  2014.  Although  the  plan  was  adequate 
with  respect  to  the  Total  Ship  Survivability  Trial  (TSST)  on 
CVN-78  and  the  Analytical  Bridge,  DOT&E  returned  the 
plan  to  the  Navy  because  it  called  for  the  F nil  Ship  Shock 
Trial  (FSST)  on  CVN-79  instead  of  CVN-78.  The  original 
Alternative  Live  Fire  Strategy  prepared  by  the  Navy  and 
approved  by  DOT&E  on  December  9,  2008,  stated  the  FSST 
would  be  conducted  on  CVN-78.  The  Navy  unilaterally 
reneged  on  the  approved  strategy  on  June  18,  2012.  The  Navy 
has  not  submitted  an  updated  Test  and  Evaluation  Master  Plan 
(TEMP)  to  DOT&E.  The  last  approved  TEMP  was  TEMP 
1610  Revision  B,  which  was  approved  in  2007. 

•  TEMP  1610  Revision  C,  which  is  in  revision,  improves 
integrated  platform-level  developmental  testing,  reducing  the 
likelihood  that  platform-level  problems  will  be  discovered 
during  IOT&E.  In  addition,  the  Program  Office  is  in  the 
process  of  refining  the  post-delivery  schedule  to  further 
integrate  testing. 

•  CVN-78  incorporates  newly  designed  catapults,  arresting  gear, 
weapons  elevators,  and  radar,  which  are  all  critical  for  flight 
operations. 

•  Reliability  for  the  catapidt  and  arresting  gear  systems  have 
not  been  reported  on  in  over  a  year.  Before  the  Navy  stopped 
tracking/reporting  on  catapult  and  arresting  gear  performance, 
both  systems  were  performing  well  below  their  projected 
target  to  achieve  required  reliability.  Reliability  test  data 

are  not  available  for  the  radar  and  the  weapons  elevators. 
DOT&E  assesses  that  the  poor  or  unknown  reliability  of  these 
critical  systems  will  be  the  most  significant  risk  to  CVN-78’s 
successful  completion  of  IOT&E. 

•  Testing  at  the  Electromagnetic  Aircraft  Launching  System 
(EMALS)  functional  demonstration  test  site  at  Joint  Base 
McGuire- Dix-Lakehurst,  New  Jersey,  discovered  an  excessive 
EMALS  holdback  release  dynamics  during  F/A-18E/F  and 
EA-18G  catapult  launches  with  wing-mounted  480-gallon 
external  fuel  tanks  (EFTs).  This  discovery,  if  uncorrected, 
would  preclude  the  Navy  from  conducting  normal  operations 
of  the  F/A-18E/F  and  EA-18G  from  CVN-78.  The  Navy  has 
no  plan  to  address  this  discovery  in  FY15. 

•  The  CVN-78  design  is  intended  to  reduce  manning.  As 
manning  requirements  have  been  further  developed,  analysis 
indicates  the  present  design  has  insufficient  berthing  for 
some  ranks  requiring  re-designation/redesign  of  some  spaces 


as  a  possible  solution.  The  ship  will  not  be  delivered  with 
sufficient  empty  berthing  for  the  CVN-78’s  Service  Life 
Allowance  (SLA).  The  SLA  would  provide  empty  bunks  to 
allow  for  changes  in  the  crew  composition  over  CVN-78’s 
expected  50-year  lifespan,  as  well  as  surge  capacity,  and  ship 
riders  for  repairs,  assists,  and  inspections. 

•  The  CVN-78  combat  system  for  self-defense  is  derived  from 
the  combat  system  on  current  carriers  and  is  expected  to  have 
similar  capabilities  and  limitations. 

•  The  Navy  continues  to  work  on  integration  challenges  related 
to  the  F-35  Joint  Strike  Fighter  (F-35)  and  its  fleet  of  aircraft 
carriers,  including  CVN-78. 

•  It  is  unlikely  that  CVN-78  will  achieve  its  Sortie  Generation 
Rate  (SGR)  (number  of  aircraft  sorties  per  day)  requirement. 
The  target  threshold  is  based  on  unrealistic  assumptions 
including  fair  weather  and  unlimited  visibility,  and  that  aircraft 
emergencies,  failures  of  shipboard  equipment,  ship  maneuvers, 
and  manning  shortfalls  will  not  affect  flight  operations. 
Discovery  of  EMALS  excessive  holdback  release  dynamics, 
as  well  as  possible  solutions,  could  significantly  limit  the 
carriers’  SGR.  DOT&E  plans  to  assess  CVN-78  performance 
during  IOT&E  by  comparing  to  the  demonstrated  performance 
of  the  Nimitz  class  carriers  as  well  as  to  the  SGR  requirement. 

•  Although  CVN-78  will  include  a  new  Heavy  underway 
replenishment  (UNREP)  system  that  will  transfer  cargo  loads 
of  up  to  12,000  pounds,  the  Navy  plans  to  install  Heavy 
UNREP  systems  on  resupply  ships  beginning  in  FY21  with 
T-AO(X). 

•  The  Navy  began  CVN-78  construction  in  2008,  and  the  ship 
was  christened  November  9,  2013.  The  schedule  to  deliver  the 
ship  has  slipped  from  September  2015  to  March  2016.  The 
development,  construction,  and  testing  of  EMALS,  Advanced 
Arresting  Gear  (AAG),  Dual  Band  Radar  (DBR),  and 
Integrated  Warfare  System  will  continue  to  drive  the  timeline. 
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System 

•  The  CVN-78  Gerald  R.  Ford  class  nuclear  aircraft  carrier 
program  is  a  new  class  of  nuclear-powered  aircraft  carriers  that 
replaces  the  previous  CVN-21  program  designation.  It  has 
the  same  hull  form  as  the  CVN-68  Nimitz  class,  but  many  ship 
systems,  including  the  nuclear  plant  and  the  flight  deck,  are 
new. 

•  The  newly  designed  nuclear  power  plant  is  intended  to  operate 
at  a  reduced  manning  level  that  is  50  percent  of  a  CVN-68 
class  ship  and  produce  significantly  more  electricity. 

•  The  CVN-78  will  incorporate  EMALS  (electromagnetic, 
instead  of  steam-powered)  and  AAG,  and  will  have  a 
smaller  island  with  a  DBR  (a  phased-array  radar,  which 
replaces/  combines  several  legacy  radars  used  on  current 
aircraft  carriers  serving  in  air  traffic  control  and  in  ship 
self-defense). 

•  The  Navy  intends  for  the  Integrated  Warfare  System  to 
be  adaptable  to  technology  upgrades  and  varied  missions 
throughout  the  ship’s  projected  operating  life  including 
increased  self-defense  capabilities  compared  to  current  aircraft 
carriers. 

•  The  Navy  redesigned  weapons  stowage,  handling  spaces,  and 
elevators  to  reduce  manning,  increase  safety,  and  increase 
throughput  of  weapons. 

•  CVN-78  has  design  features  intended  to  enhance  its  ability  to 
launch,  recover,  and  service  aircraft,  such  as  a  slightly  larger 
flight  deck,  dedicated  weapons  handling  areas,  and  increased 
aircraft  refueling  stations.  The  Navy  set  the  SGR  requirement 


for  CVN-78  to  increase  the  sortie  generation  capability  of 
embarked  aircraft  to  160  sorties  per  day  (12-hour  fly  day)  and 
to  surge  to  270  sorties  per  day  (24-hour  fly  day)  as  compared 
to  the  CVN-68  Nimitz  class  SGR  demonstration  of  120  sorties 
per  day/240  sorties  for  24-hour  surge. 

•  The  Consolidated  Afloat  Networks  and  Enterprise  Services 
(CANES)  program  replaces  five  shipboard  legacy  network 
programs  to  provide  a  common  computing  environment  for 
command,  control,  intelligence,  and  logistics. 

•  CVN-78  is  intended  to  support  the  F-35  and  future  weapons 
systems  over  the  expected  50-year  ship’s  lifespan. 

•  The  Navy  plans  to  declare  CVN-78  Initial  Operational 
Capability  in  FY17  and  achieve  Full  Operational  Capability  in 
FY19  (during  IOT&E  and  after  the  Type  Commander  certifies 
that  CVN-78  is  Major  Combat  Operations  Ready). 

Mission 

Carrier  Strike  Group  Commanders  will  use  the  CVN-78  to: 

•  Conduct  power  projection  and  strike  warfare  missions  using 
embarked  aircraft 

•  Provide  force  and  area  protection 

•  Provide  a  sea  base  as  both  a  command  and  control  platform 
and  an  air-capable  unit 

Major  Contractor 

Huntington  Ingalls  Industries,  Newport  News 

Shipbuilding  -  Newport  News,  Virginia 


Activity 

Test  Planning 

•  The  CVN-78  Gerald  R.  Ford  class  carrier  Program  Office 
revised  the  TEMP  to  align  planned  developmental  tests 
with  corresponding  operational  test  phases  and  to  identify 
platform-level  developmental  testing. 

•  The  Program  Office  released  an  updated  Post  Delivery  Test 
and  Trials  schedule. 

•  The  Navy  continues  to  develop  the  CVN-78  SGR  test 
modeling.  The  Navy  is  conducting  weekly  Configuration 
Review  Board  meetings  to  refine  requirements  for  model 
development  through  FY17.  The  ship’s  SGR  requirement 
is  based  on  a  30-plus-day  wartime  scenario.  The  Navy 
intends  to  update  the  wartime  scenario.  The  Navy  designed 
a  test  to  demonstrate  the  SGR  with  6  consecutive  12-hour 
fly  days  followed  by  2  consecutive  24-hour  fly  days.  This 
live  testing  will  be  supplemented  with  modeling  and 
simulation  from  the  Virtual  Carrier  (VCVN)  model  to 
extrapolate  results  to  the  30-plus-day  SGR  requirement. 
DOT&E  concurs  with  this  approach. 

EMALS 

•  The  EMALS  functional  demonstration  test  site  at  Joint 
Base  McGuire-Dix-Lakehurst,  New  Jersey,  continues  to 
test  the  new  electromagnetic  catapult  system.  Aircraft 


compatibility  testing  was  completed  in  April  2014.  A  total 
of  452  aircraft  launches  were  conducted  using  EA-18G, 
F/A-18E,  F/A-18C,  E-2D,  T-45,  and  C-2A  aircraft.  The 
testing  discovered  excessive  EMALS  holdback  release 
dynamics  during  F/A-18E/F  and  EA-18G  catapult  launches 
with  wing-mounted  480-gallon  EFTs.  Aircraft  dynamics 
are  considered  excessive  if  they  exceed  stress  limits  of 
the  airframe,  internal,  or  external  stores.  This  discovery, 
if  uncorrected,  would  preclude  normal  employment  of 
the  F/A-18E/F  and  EA-18G  from  CVN-78.  There  is  no 
funding  at  this  time  to  correct  this  deficiency. 

•  The  Navy  has  also  conducted  over  3,000  dead-load 
launches  (non-aircraft,  weight  equivalent,  and  simulated 
launches).  EMALS  is  currently  undergoing  laboratory 
environmental  qualification  testing  and  testing  of 
engineering  changes  to  correct  observed  failures. 

Shipboard  testing  began  on  August  11,  2014,  with  below 
decks  components.  Approximately  94  percent  of  the 
EMALS  equipment  has  been  delivered  to  the  shipyard. 

All  linear  motors  are  planned  to  be  installed  by  the  end  of 
1QFY15  to  include  the  main  power  cables  on  catapults 
1,  2,  and  3. 


166  CVN-78 


F Y1  4  NAVY  PROGRAMS 


AAG 

•  The  Navy  continues  testing  the  AAG  on  a  jet  car  track 
at  Joint  Base  McGuire-Dix-Lakehurst,  New  Jersey,  with 
528  arrestments  completed  by  August  2014.  Testing  has 
prompted  system  design  changes.  A  failure  in  the  water 
twister  suspended  testing  in  November  2013;  the  Navy 
authorized  return-to-test  in  January  2014.  Dead-load  testing 
resumed  in  May  to  validate  performance  of  modifications 

to  the  mechanical  brake.  Runway  Arrested  Landing  Site 
(RALS)  preparation  began  using  equipment  planned 
for  CVN-78.  The  Navy  de-scoped  the  number  4  AAG 
engine,  reducing  the  total  arresting  gear  engines  on  the 
ship,  including  the  barricade,  to  three,  and  diverted  the 
following  equipment  to  RALS  in  Lakehurst:  the  water 
twisters,  electric  motors,  purchase  cable  drum  assemblies, 
and  cable  shock  absorbers  for  the  number  4  arresting  gear 
engine.  Approximately  94  percent  of  the  remaining  AAG 
equipment  has  been  delivered  to  the  shipyard. 

CANES 

•  The  Navy  has  scheduled  developmental  and  follow-on 
operational  testing  of  the  force-level  CANES  configuration 
used  on  the  Nimitz  and  Gerald  R.  Ford  classes  for  2Q  and 
3QFY15. 

•  The  Navy  conducted  operational  testing  in  accordance  with 
a  DOT&E-approved  test  plan. 

•  The  Navy  conducted  integrated  testing  and  IOT&E  of  the 
unit-level  Aegis  destroyer  configuration  in  3Q  and  4QFY14. 

DBR 

•  The  Navy  installed  a  production  Multi-Function  Radar 
and  reactivated  the  Engineering  Development  Model  of 
the  Volume  Search  Radar  at  the  Surface  Combat  System 
Center  at  Wallops  Island,  Virginia.  The  Navy  planned 
to  begin  testing  in  January  2013;  however,  the  testing 
has  slipped  repeatedly,  and  to  date,  no  live  testing  with 
the  full  production  DBR  has  been  completed.  The  first 
government-led  integrated  test  events  began  in  1QFY14. 

The  first  developmental  testing  of  DBR  began  in  4QFY14 
at  Wallops  Island. 

Manning 

•  The  Navy  conducted  CVN-78  Manning  War  Game  III 
in  July  2014  to  identify  CVN-78  unique  Manpower, 
Personnel,  Training,  and  Education  planning  and  execution 
concerns.  The  results  of  the  War  Game  have  not  been 
published. 

JPALS 

•  The  Joint  Precision  Approach  and  Landing  System  (JPALS) 
is  no  longer  funded  for  CVN-78. 

•  In  June  2014,  following  a  Nunn-McCurdy  breach, 
USD(AT&L)  rescinded  Milestone  B  approval  for  the 
sea-based  Increment  1 A  of  the  JPALS  land-  and  sea-based 
multiple-increment  JPALS  program.  USD(AT&L)  directed 
the  Navy  to  restructure  the  multiple-increment  program 
into  a  single  increment  focusing  on  sea-based  requirements 
primarily  supporting  F-35  and  future  Unmanned 
Carrier- Launched  Airborne  Surveillance  and  Strike 
(UCLASS)  aircraft. 


•  Under  the  restructured  program,  the  Navy  will  complete 
the  development  phase  for  precision  approach  and  landing 
capability  for  JPALS-equipped  manned  aircraft  at  sea 
with  the  addition  of  risk  reduction  efforts  to  prepare  for 
future  manned  and  unmanned  auto-land  capability.  The 
actual  production  phase  of  JPALS  will  be  deferred  until  it 
is  required  for  F-35  and  UCLASS  aircraft.  There  will  be 
no  retrofitting  of  legacy  aircraft  with  JPALS.  The  Navy 
will  need  to  maintain  both  the  legacy  approach  and  landing 
system  and  JPALS  onboard  each  ship  designated  to  receive 
future  JPALS-equipped  aircraft. 

F-35 

•  The  Navy  is  working  to  address  several  F-35  integration 
challenges  on  its  aircraft  carriers.  In  general,  these  issues 
affect  all  of  the  Navy’s  carriers,  not  just  CVN-78. 

•  In  FY12,  a  test  of  the  F-35  arresting  hook  system  identified 
problems  with  the  design.  After  failing  to  engage  the 
arresting  cable  and  demonstrating  insufficient  load-carrying 
capacity,  the  Navy  has  redesigned  the  arresting  hook 
system.  Testing  at  Joint  Base  McGuire-Dix-Lakehurst 
began  in  April  2014  and  completed  in  September  2014, 
followed  by  shipboard  trials  onboard  USS  Nimitz  in 
1QFY15.  The  re-designed  hook  has  been  successful  in 
arresting  the  aircraft.  The  Navy  is  redesigning  the  cooling 
system  in  the  CVN  78’s  Jet  Blast  Deflectors  (JBDs).  The 
JBDs  deflect  engine  exhaust  during  catapult  launches. 

The  redesign  is  needed  to  handle  F-35  engine  exhaust  and 
will  include  improvements  in  cooling  flow  and  eventual 
addition  of  side-cooling  panels.  Until  side-cooling  panels 
are  installed,  the  F-35  will  be  thrust  and  weight  limited  for 
take-off,  with  associated  penalties  in  payload  and/or  range. 
Side  cooling  panels  are  expected  to  be  installed  on  CVNs  in 
the  early  2020 ’s. 

•  CVN-78  will  receive  the  new  Heavy  UNREP  system.  To 
use  the  Heavy  UNREP  capability,  both  the  carrier  and  the 
resupply  ship  must  be  equipped  with  the  system.  This 
new  Heavy  UNREP  system,  along  with  heavy  vertical  lift 
aircraft  not  embarked  on  carriers,  are  the  only  systems 
currently  capable  of  resupplying  the  F-35  containerized 
engine  while  the  carrier  is  underway.  Today,  only  one 
combat  logistic  ship  has  Heavy  UNREP,  USNS  Arctic.  The 
Navy  plans  to  have  Heavy  UNREP  systems  installed  on 
resupply  ships  starting  with  T-AO(X)  in  FY21.  The  current 
acquisition  strategy  has  one  T-AO  ship  delivery  every  year 
after  that  for  a  total  of  1 7  ships. 

•  The  Navy  is  designing  separate  charging  and  storage 
lockers  for  the  lithium-ion  batteries  required  for  the  F-35 
and  for  F-35  weapons  loading  support  equipment.  This 
includes  aircraft  battery  accommodations  below  decks 
and  ready  service  lockers  for  weapons  loader  batteries  on 
the  flight  deck.  The  Navy  is  also  designing  a  new  storage 
locker  for  pilot  flight  equipment  as  the  F-35  helmet  is  larger 
and  more  fragile  than  legacy  helmets. 

•  The  Navy  has  completed  F-35  cyclic  thermal  strain  testing 
and  concluded  that  repeated  F-35  sortie  generation  at 
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combat  rated  thrust,  i.e.,  afterburner,  will  not  cause  cyclic 
thermal  strain  on  the  CVN-78  flight  deck  structure. 

•  Unlike  current  fleet  aircraft,  the  F-35  carries  ordnance 
in  internal  bays.  This  will  require  changes  to  aircraft 
firefighting  techniques  for  the  F-35.  The  Navy  is 
developing  new  firefighting  equipment  to  attach  to  existing 
hose  nozzles,  including  a  tow-bar  mounted  spray  device 
for  open  bay  firefighting,  and  a  penetrating  device  to  punch 
through  the  aircraft  skin  for  closed  bay  scenarios. 

•  The  F-35  Joint  Program  Office  has  initiated  a  tire  redesign 
for  the  F-35B  due  to  higher  than  predicted  wear  rates.  The 
Navy  has  not  yet  settled  on  a  strategy  for  dealing  with  a 
possible  higher  tire  storage  requirement. 

•  The  F-35  is  very  loud  aircraft.  The  noise  level  in  some 
operating  envelopes  is  presently  being  tested  on  the  flight 
deck.  The  Navy  has  determined  the  problem  of  aircraft 
noise  on  the  flight  deck  is  significantly  worse  than  they 
originally  thought.  This  may  require  the  installation  of 
noise  abatement  material  below  the  flight  deck  to  allow  for 
conversational  speech  in  work  spaces  located  on  the  03 
level.  The  Navy  plans  to  investigate  noise  levels  during 
shipboard  trials  in  1QFY15  to  help  determine  a  solution. 

•  The  F-35  ejection  seat  has  a  higher  center  of  gravity  than 
legacy  seats,  requiring  additional  tie  downs  for  heavy  seas 
when  installed  in  the  maintenance  dolly.  The  F-35  program 
is  planning  to  use  the  Navy  Aircrew  Common  Ejection  Seat 
(NACES)  Seat  Dolly  and  will  provide  an  adaptor  for  its 
seat.  The  Navy  and  F-35  Program  Office  will  assess  the 
need  for  changes  to  the  seat  shop  when  the  drawings  for  the 
adaptor  are  completed.  The  use  of  the  NACES  Seat  Dolly 
is  anticipated  to  eliminate  the  need  to  change  the  seat  shop. 
The  Navy  is  currently  determining  what  modifications  this 
will  require  for  the  seat  shop. 

LFT&E 

•  The  Navy  submitted  the  LFT&E  Management  Plan, 
Revision  B  in  July  2014.  Although  the  plan  was  adequate 
with  respect  to  the  TSST  on  CVN-78  and  the  Analytical 
Bridge,  DOT&E  returned  the  plan  to  the  Navy  because 

it  called  for  the  FSST  on  CVN-79  instead  of  CVN-78 
as  stipulated  in  the  original  Alternate  Live  Fire  Strategy 
approved  on  December  9,  2008. 

Assessment 

Test  Planning 

•  A  new  TEMP  is  under  development  to  address  problems 
with  the  currently-approved  TEMP.  The  TEMP  in  the 
approval  process  improves  integrated  platform-level 
developmental  testing,  reducing  the  likelihood  that 
platform-level  problems  will  be  discovered  during  IOT&E. 
In  addition,  the  Program  Office  is  in  the  process  of  refining 
the  post-delivery  schedule  to  further  integrate  testing. 

•  The  current  state  of  the  VCVN  model  does  not  fully  provide 
for  an  accurate  accounting  of  SGR  due  to  a  lack  of  fidelity 
regarding  manning  and  equipment/aircraft  availability. 

Spiral  development  of  the  VCVN  model  continues  in  order 


to  ensure  that  the  required  fidelity  will  be  available  to 
support  the  SGR  assessment  during  IOT&E. 

•  The  Navy  plans  to  take  delivery  CVN-78  in  March  2016. 
The  ship’s  post-shipyard  shakedown  availability  will  follow 
delivery  in  late  2016.  During  the  post-shipyard  shakedown 
availability  installations  of  some  systems  will  be  completed. 
The  first  at-sea  operational  test  and  evaluation  of  CVN-78 
will  begin  in  September  2017. 

Reliability 

•  CVN-78  includes  several  systems  that  are  new  to  aircraft 
carriers;  four  of  these  systems  stand  out  as  being  critical  to 
flight  operations:  EMALS,  AAG,  DBR,  and  the  Advanced 
Weapons  Elevators  (AWEs).  Overall,  the  uncertain 
reliability  of  these  four  systems  is  the  most  significant  risk 
to  the  CVN-78  IOT&E.  All  four  of  these  systems  will  be 
tested  for  the  first  time  in  their  shipboard  configurations 
aboard  CVN-78.  Reliability  estimates  derived  from  test 
data  were  provided  last  year  for  EMALS  and  AAG  and 
are  discussed  below.  The  Navy  has  stated  that  in  the  last 
year,  they  did  not  assess  EMALS  and  AAG  reliability  due 
to  systems’  redesign  and  investigative  and  developmental 
testing.  For  DBR  and  AWE,  estimates  based  on  test  data 
are  not  available  and  only  engineering  reliability  estimates 
are  available. 

EMALS 

•  EMALS  is  one  of  the  four  systems  critical  to  flight 
operations.  While  testing  to  date  has  demonstrated  that 
EMALS  should  be  able  to  launch  aircraft  planned  for 
CVN-78’s  air  wing,  present  limitations  on  F/A-18E/F 
and  EA-18G  configurations  as  well  as  the  system’s 
reliability  remains  uncertain.  As  of  December  2013,  at  the 
Lakehurst,  New  Jersey,  test  site,  over  1,967  launches  had 
been  conducted  with  20 1  chargeable  failures.  At  that  time, 
the  program  estimates  that  EMALS  has  approximately 
240  Mean  Cycles  Between  Critical  Failure  in  the  shipboard 
configuration,  where  a  cycle  represents  the  launch  of  one 
aircraft.  Based  on  expected  reliability  growth,  the  failure 
rate  for  the  last  reported  Mean  Cycles  Between  Critical 
Failure  was  five  times  higher  than  should  have  been 
expected.  As  of  August  2014,  the  Navy  has  reported  that 
over  3,017  launches  have  been  conducted  at  the  Lakehurst 
test  site,  but  have  not  provided  DOT&E  with  an  update  of 
failures.  The  Navy  intends  to  provide  DOT&E  an  update  of 
failures  in  December  2014. 

AAG 

•  AAG  is  another  system  critical  to  flight  operations.  Testing 
to  date  has  demonstrated  that  AAG  should  be  able  to 
recover  aircraft  planned  for  the  CVN-78  air  wing,  but 

as  with  EMALS,  AAG’s  reliability  is  uncertain.  At  the 
Lakehurst  test  site,  7 1  arrestments  were  conducted  early 
in  2013  and  9  chargeable  failures  occurred.  The  Program 
Office  last  provided  reliability  data  in  December  2013  and 
estimated  that  AAG  had  approximately  20  Mean  Cycles 
Between  Operational  Mission  Failure  in  the  shipboard 
configuration,  where  a  cycle  represents  the  recovery  of 
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one  aircraft.  Following  these  tests,  the  Navy  modified  the 
system  and  has  yet  to  score  reliability  of  AAG.  Based  on 
expected  reliability  growth  as  of  2013,  the  failure  rate  was 
248  times  higher  than  should  have  been  expected. 

DBR 

•  Previous  testing  of  Navy  combat  systems  similar  to 
CVN-78’s  revealed  numerous  integration  problems  that 
degrade  the  performance  of  the  combat  system.  Many 
of  these  problems  are  expected  to  exist  on  CVN-78.  The 
previous  results  emphasize  the  necessity  of  maintaining  a 
DBR/CVN-78  combat  system  asset  at  Wallops  Island.  The 
Navy  is  considering  long-term  plans  (i.e.,  beyond  FY15) 
for  testing  DBR  at  Wallops  Island,  but  it  is  not  clear  if 
resources  and  funding  will  be  available.  Such  plans  are 
critical  to  delivering  a  fully-capable  combat  system  and 
ensuring  life-cycle  support  after  CVN-78  delivery  in  2016. 

SGR 

•  It  is  unlikely  that  CVN-78  will  achieve  its  SGR 
requirement.  The  target  threshold  is  based  on  unrealistic 
assumptions  including  fair  weather  and  unlimited  visibility, 
and  that  aircraft  emergencies,  failures  of  shipboard 
equipment,  ship  maneuvers,  and  manning  shortfalls  will  not 
affect  flight  operations.  DOT&E  plans  to  assess  CVN-78 
performance  during  IOT&E  by  comparing  it  to  the  SGR 
requirement  as  well  as  to  the  demonstrated  performance  of 
the  Nimitz  class  carriers. 

•  During  the  operational  assessment,  DOT&E  conducted  an 
analysis  of  past  aircraft  carrier  operations  in  major  conflicts. 
The  analysis  concludes  that  the  CVN-78  SGR  requirement 
is  well  above  historical  levels  and  that  CVN  78  is  unlikely 
to  achieve  that  requirement.  There  are  concerns  with  the 
reliability  of  key  systems  that  support  sortie  generation  on 
CVN-78.  Poor  reliability  of  these  critical  systems  could 
cause  a  cascading  series  of  delays  during  flight  operations 
that  would  affect  CVN-78’s  ability  to  generate  sorties,  make 
the  ship  more  vulnerable  to  attack,  or  create  limitations 
during  routine  operations.  DOT&E  assesses  the  poor  or 
unknown  reliability  of  these  critical  subsystems  will  be  the 
most  significant  risk  to  CVN-78’s  successful  completion 

of  IOT&E.  The  analysis  also  considered  the  operational 
implications  of  a  shortfall  and  concluded  that  as  long  as 
CVN-78  is  able  to  generate  sorties  comparable  to  Nimitz 
class  carriers,  the  operational  implications  of  CVN-78  will 
be  similar  to  that  of  a  Nimitz  class  carrier. 

Manning 

•  Current  manning  estimates  have  shortages  of  bunks 
for  Chief  Petty  Officers  (CPOs)  and  do  not  provide  the 
required  10  percent  SLA  for  all  berthing.  The  Navy  plans 
to  re-designate/design  some  officer  rooms  as  CPO  berthing 
spaces.  Per  the  Office  of  the  Chief  of  Naval  Operations 
Instruction  9640. IB,  Shipboard  Habitability  Program, 

all  new  ships  are  required  to  have  a  growth  allowance  of 
10  percent  of  the  ship’s  company  when  the  ship  delivers. 
The  SLA  provides  empty  bunks  to  allow  for  changes  in  the 
crew  composition  over  CVN  78’s  expected  50-year  lifespan 


and  provides  berthing  for  visitors  and  Service  members 
temporarily  assigned  to  the  ship. 

JPALS 

•  As  the  Navy  reformulates  the  JPALS  Test  and  Evaluation 
Master  Plan,  it  faces  significant  challenges  in  defining 
how  it  will  demonstrate  the  operational  effectiveness  and 
operational  suitability  of  the  restructured  system  without  a 
representative  aircraft  platform. 

F-35 

•  The  arresting  hook  system  remains  an  integration  risk  as  the 
F-35  development  schedule  leaves  no  time  for  discovering 
new  problems.  The  redesigned  tail  hook  has  an  increased 
downward  force  as  well  as  sharper  design  that  may  induce 
greater  than  anticipated  wear  on  the  flight  deck. 

•  F-35  noise  levels  remain  moderate  to  high  risk  in  F-35 
integration  and  will  require  modified  carrier  flight  deck 
procedures. 

-  Flight  operations  normally  locate  some  flight  deck 
personnel  in  areas  where  double  hearing  protection 
would  be  insufficient  during  F-35  operations.  To 
partially  mitigate  noise  concerns,  the  Navy  will  procure 
new  hearing  protection  with  active  noise  reduction  for 
flight  deck  personnel. 

-  Projected  noise  levels  one  level  below  the  flight  deck 
(03  level),  which  includes  mission  planning  spaces,  will 
require  at  least  single  hearing  protection  that  will  make 
mission  planning  difficult.  The  Navy  is  working  to 
mitigate  the  effects  of  the  increased  noise  levels  adjacent 
to  the  flight  deck. 

•  Storage  of  the  F-35  engine  is  limited  to  the  hangar  bay, 
which  will  affect  hangar  bay  operations.  The  impact  on  the 
F-35  logistics  footprint  is  not  yet  known. 

•  Lightning  protection  of  F-35  aircraft  while  on  the  flight 
deck  will  require  the  Navy  to  modify  nitrogen  carts  to 
increase  their  capacity.  Nitrogen  is  filled  in  fuel  tank 
cavities  while  aircraft  are  on  the  flight  deck  or  hangar  bay. 

•  F-35  remains  unable  to  share  battle  damage  assessment 
and  non-traditional  Intelligence,  Surveillance,  and 
Reconnaissance  information  captured  on  the  aircraft 
portable  memory  device  or  cockpit  voice  recorder  in 
real-time.  In  addition,  the  CVN-78  remains  unable  to 
receive  and  display  imagery  transmitted  through  Link  16 
because  of  bandwidth  limitations;  this  problem  is  not 
unique  to  F-35.  These  capability  gaps  were  identified 

in  DOT&E ’s  FY12  Annual  Report.  The  Combatant 
Commanders  have  requested  these  capabilities  to  enhance 
decision-making. 

LFT&E 

•  The  Navy  has  made  substantial  progress  on  defining  the 
scope  of  the  TSST  and  the  Analytical  Bridge  task.  While 
these  portions  of  the  LFT&E  Management  Plan  were 
adequately  defined  in  the  Revision  B  document,  DOT&E 
returned  the  LFT&E  Management  Plan  to  the  Navy  solely 
on  the  basis  of  the  FSST  on  CVN  79  verses  CVN-78. 


CVN-78 
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•  CVN-78  has  many  new  critical  systems,  such  as  EMALS, 
AAG,  and  DBR,  that  have  not  undergone  shock  trials  on 
other  platforms.  Unlike  past  tests  on  other  new  classes  of 
ships  with  legacy  systems,  the  performance  of  CVN-78’s 
new  critical  systems  is  unknown. 

•  The  Navy  proposes  delaying  the  shock  trial  by  five  to 
seven  years  because  of  the  approximately  four-  to 

six- month  delay  required  to  perform  the  FSST.  The  benefit 
of  having  test  data  to  affect  the  design  of  future  carriers  in 
the  class  outweighs  the  delay  in  delivery  of  CVN-78  to  the 
fleet  to  conduct  this  test.  The  delay  is  not  a  sufficient  reason 
to  postpone  the  shock  trial. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  should 
continue  to  address  the  eight  remaining  FY10,  FY11,  and 
FY13  recommendations. 

1 .  Adequately  test  and  address  integration  challenges  with 
F-35;  specifically: 

-  Logistics  (unique  concerns  for  storage  and  transportation) 

-  Changes  required  to  JBDs 

-  Changes  to  flight  deck  procedures  due  to  heat  and  noise 

-  Autonomic  Logistics  Information  System  integration 

2.  Finalize  plans  that  address  CVN-78  Integrated  Warfare 
System  engineering  and  ship’s  self-defense  system 
discrepancies  prior  to  the  start  of  IOT&E. 

3.  Continue  aggressive  EMALS  and  AAG  risk-reduction 
efforts  to  maximize  opportunity  for  successful  system 


design  and  test  completion  in  time  to  meet  required  in-yard 
dates  for  shipboard  installation  of  components. 

4.  Continue  development  of  a  realistic  model  for  determining 
CVN-78’s  SGR,  while  utilizing  realistic  assumptions 
regarding  equipment  availability,  manning,  and  weather 
conditions  for  use  in  the  IOT&E. 

5.  Provide  scheduling,  funding,  and  execution  plans  to 
DOT&E  for  the  live  SGR  test  event  during  the  IOT&E. 

6.  Continue  to  work  with  the  Navy’s  Bureau  of  Personnel  to 
achieve  adequate  depth  and  breadth  of  required  personnel 
to  sufficiently  meet  Navy  Enlisted  Classification  fit/fill 
manning  requirements  of  CVN-78. 

7.  Conduct  system-of-systems  developmental  testing  to 
preclude  discovery  of  deficiencies  during  IOT&E. 

8.  Address  the  uncertain  reliability  of  EMALS,  AAG,  DBR, 
and  AWE.  These  systems  are  critical  to  CVN-78  flight 
operations,  and  are  the  largest  risk  to  the  program. 

FY 14  Recommendations.  The  Navy  should: 

1 .  Aggressively  fund  and  address  a  solution  for  the  excessive 
EMALS  holdback  release  dynamics  during  F/A-18E/F  and 
EA-18G  catapult  launches  with  wing-mounted  480-gallon 
EFTs. 

2.  Plan  for  fully  integrated,  robust,  end-to-end  testing  of  the 
restructured  JPALS  onboard  both  manned  high-performance 
and  unmanned  aircraft,  including  operations  in  neutral  and 
potentially  hostile  electronic  warfare  environments. 
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DDG  51  Flight  III  Destroyer/Air  and  Missile  Defense 
Radar  (AMDR)/Aegis  Modernization 


Executive  Summary 

•  On  March  6,  2014,  the  Deputy  Secretary  of  Defense 
(DEPSECDEF)  issued  a  Resource  Management  Decision 
memorandum  directing  the  Navy  to  develop  a  plan  to  conduct 
at-sea  testing  of  the  self-defense  capability  of  the  DDG  51 
Flight  III  Destroyer  with  the  Air  and  Missile  Defense  Radar 
(AMDR)  and  Aegis  Combat  System.  The  plan  was  to  be 
approved  by  DOT&E  and  then  adequately  funded  by  the 
Navy.  However,  the  Navy  has  not  provided  any  plan  to 
DOT&E  or  planned  funding  to  facilitate  the  testing. 

•  On  April  23,  2014,  DOT&E  issued  a  memorandum  to 
USD(AT&L)  stating  the  intention  to  not  approve  any 
operational  test  plan  for  an  Early  Operational  Assessment 
(EOA)  of  the  AMDR  due  to  non-availability  of  the  required 
AMDR  hardware  and  software. 

•  On  September  10,  2014,  DOT&E  issued  a  classified 
memorandum  to  USD(AT&L)  with  a  review  of  the  Navy 
Program  Executive  Office  for  Integrated  Warfare  Systems 
Design  of  Experiments  study.  The  study  attempted  to  provide 
a  technical  justification  to  show  the  test  program  did  not 
require  using  a  Self-Defense  Test  Ship  (SDTS)  to  adequately 
assess  the  self-defense  capability  of  the  DDG  51  Flight  III 
Class  Destroyers.  DOT&E  found  the  study  presented  a 
number  of  flawed  rationales,  contradicted  itself,  and  failed  to 
make  a  cogent  argument  for  why  an  SDTS  is  not  needed  for 
operational  testing. 

System 

•  The  DDG  5 1  Flight  III  Destroyer  will  be  a  combatant  ship 
equipped  with  the: 

-  AMDR  three-dimensional  (range,  altitude,  and  azimuth) 
multi-function  radar 

-  Aegis  Combat  System 

-  AN/SQQ-89  Undersea  Warfare  suite  that  includes  the 
AN/SQS-53  sonar 

-  MH-60R  helicopter 

-  Close-In  Weapon  System 

-  Five-inch  diameter  gun 

-  Vertical  Launch  System  that  can  launch  Tomahawk, 
Standard  Missiles  (SM-2,  -3,  and  -6),  and  Evolved 
SeaSparrow  Missiles  (ESSMs) 

•  The  Navy  is  developing  the  AMDR  to  provide  simultaneous 
sensor  support  of  integrated  air  and  missile  defense  (IAMD) 
and  air  defense  (including  self-defense)  missions.  IAMD  and 
air  defense  require  extended  detection  ranges  and  increased 
radar  sensitivity  against  advanced  threats  with  high  speeds  and 
long  interceptor  fly-out  times.  The  three  major  components  of 
AMDR  are: 


-  The  AMDR  S-band  radar  will  provide  IAMD,  as  well  as 
search,  track,  cueing,  missile  discrimination,  air  defense 
non-cooperative  target  recognition,  S-band  missile 
communications,  surveillance  capability  for  ship  self 
defense  and  area  air  defense,  and  S-band  kill  assessment 
support  functions. 

-  The  AMDR  X-band  radar  (AMDR-X)  will  provide  horizon 
and  surface  search  capabilities  in  addition  to  navigation 
and  periscope  detection/discrimination  functions.  The 
Navy  is  delaying  development  of  the  AMDR-X.  The 
existing  AN/SPQ-9B  radar  will  provide  these  X-band 
functions  in  the  interim. 

-  The  Radar  Suite  Controller  will  provide  radar  resource 
management  and  coordination,  as  well  as  an  open  interface 
with  the  ship  combat  system. 

•  The  Aegis  Combat  System  is  an  integrated  naval  weapons 
system  that  uses  computers  and  radars  to  form  an  advanced 
command  and  decision  capability,  and  a  weapon  control 
system  to  track  and  guide  weapons  to  destroy  enemy  targets. 

-  The  Navy’s  Aegis  Modernization  program  is  a  planned, 
phased  program  that  provides  updated  technology  and 
combat  systems  for  existing  Aegis-guided  missile  cruisers 
(CG  47)  and  destroyers  (DDG  51)  as  well  as  the  DDG  51 
Flight  III  Destroyers. 

-  The  Aegis  Modernization  program  will  provide  an 
improved  Advanced  Capability  Build  combat  system 
variant  for  the  DDG  5 1  Flight  III  Destroyers  equipped  with 
the  AMDR. 
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Mission 

•  The  Navy  will  use  the  DDG  51  Flight  III  Destroyer  equipped 
with  the  Aegis  Combat  System  and  AMDR  to  provide  joint 
battlespace  threat  awareness  and  defense  capability  to  counter 
current  and  future  threats  in  support  of  joint  forces  ashore  and 
afloat. 

•  The  Navy  will  use  the  AMDR  S-band  radar/Radar  Suite 
Controller  with  the  AN/SPQ-9B  and  the  Aegis  Modernization 
Program  to  support  the  following  DDG  5 1  Flight  III  Destroyer 
missions: 

-  Area  air  defense  (to  include  self-defense)  to  counter 
advanced  air  and  cruise  missile  threats  and  increase  ship 
survivability 

-  Detect,  track,  discriminate,  and  provide  missile 
engagement  support  (including  kill  assessment)  to  counter 
ballistic  missile  threats 

-  Surface  surveillance,  precision  tracking,  and  missile  and 
gun  engagements  to  counter  surface  threats 


-  Undersea  Warfare  with  periscope  detection  and 
discrimination 

-  Detect  and  track  enemy  artillery  projectiles  to  support 
combat  system  localization  of  land-battery  launch  positions 
by  the  DDG  51  Flight  III  Combat  System 

-  Detect  and  track  own-ship  gun  projectiles  in  support  of 
surface  warfare  and  naval  surface  fire  support 

Major  Contractors 

•  DDG  5 1  Flight  III  Destroyer:  General  Dynamics  Marine 
Systems  Bath  Iron  Works  -  Bath,  Maine 

•  AMDR:  Raytheon  -  Sudbury,  Massachusetts 

•  Aegis  Combat  System:  Lockheed  Martin  Marine  Systems  and 
Sensors  -  Moorestown,  New  Jersey 


Activity 

•  On  March  6,  2014,  DEPSECDEF  issued  a  Resource 
Management  Decision  memorandum  directing  the  Navy  to 
develop  a  plan  to  conduct  at-sea  testing  of  the  self-defense 
capability  of  the  DDG  5 1  Flight  III  Destroyer  with  the  AMDR 
and  Aegis  Combat  System.  The  plan  was  to  be  approved  by 
DOT&E  and  then  adequately  funded  by  the  Navy.  To  date, 
the  Navy  has  not  provided  any  plan  to  DOT&E  or  funding  in 
response  to  this  direction. 

•  On  April  23,  2014,  DOT&E  issued  a  memorandum  to 
USD(AT&L)  stating  the  operational  test  plan  for  an  EOAof 
the  AMDR  could  not  be  approved  because  the  required  AMDR 
hardware  and  software  were  not  available  as  planned,  per 

the  2010  DOT&E-  and  Navy-approved  Test  and  Evaluation 
Strategy,  and  as  briefed  to  the  Deputy  Assistant  Secretary  of 
Defense  (Strategic  and  Tactical  Systems)  in  2012.  A  prototype 
AMDR  array,  coupled  to  an  upgraded  radar  controller  using 
basic  software  for  radar  control  and  simple  search  and  track 
functionality,  was  expected  to  be  available.  The  lack  of  this 
hardware  and  software  would  have  limited  the  EOA  to  a 
“table-top”  review  of  program  documentation,  program  plans, 
and  available  design  data,  which  would,  in  DOT&E’s  view, 
not  have  been  a  worthwhile  use  of  resources. 

•  On  September  10,  2014,  DOT&E  issued  a  classified 
memorandum  to  USD(AT&L)  that  provided  a  review  of 
the  Navy  Program  Executive  Office  for  Integrated  Warfare 
Systems  Design  of  Experiments  study.  The  study  attempted 
to  provide  a  technical  justification  to  show  the  test  program 
did  not  require  using  an  SDTS  to  adequately  assess  the 
self-defense  capability  of  the  DDG  51  Flight  III  Class 
Destroyers.  DOT&E  found  the  study  presented  a  number 
of  flawed  rationales,  contradicted  itself,  and  failed  to  make 
a  cogent  argument  for  why  an  SDTS  is  not  needed  for 
operational  testing. 


Assessment 

•  DOT&E’s  assessment  continues  to  be  that  the  operational  test 
programs  for  the  AMDR,  Aegis  Modernization,  and  DDG  5 1 
Flight  III  Destroyer  programs  are  not  adequate  to  fully  assess 
their  self-defense  capabilities  in  addition  to  being  inadequate 
to  test  the  following  Navy-approved  AMDR  and  DDG  5 1 
Flight  III  requirements. 

-  The  AMDR  Capability  Development  Document  describes 
AMDR’s  IAMD  mission,  which  requires  AMDR  to 
support  simultaneous  defense  against  multiple  ballistic 
missile  threats  and  multiple  advanced  anti-ship  cruise 
missile  (ASCM)  threats.  The  Capability  Development 
Document  also  includes  an  AMDR  minimum  track  range 
Key  Performance  Parameter. 

-  The  DDG  5 1  Flight  III  Destroyer  has  a  survivability 
requirement  directly  tied  to  meeting  a  self-defense 
requirement  threshold  against  ASCMs  described  in  the 
Navy’s  Surface  Ship  Theater  Air  and  Missile  Defense 
Assessment  document  of  July  2008.  It  clearly  states 
that  area  defense  will  not  defeat  all  the  threats,  thereby 
demonstrating  that  area  air  defense  will  not  completely 
attrite  all  ASCM  raids  and  that  individual  ships  must  be 
capable  of  defeating  ASCM  leakers  in  the  self-defense 
zone. 

•  Use  of  manned  ships  for  operational  testing  with  threat 
representative  ASCM  surrogates  in  the  close-in,  self-defense 
battlespace  is  not  possible  due  to  Navy  safety  restrictions 
because  targets  and  debris  from  intercepts  pose  an 
unacceptable  risk  to  personnel  at  ranges  where  some  of  the 
engagements  will  take  place.  The  November  2013  mishap 
on  the  USS  Chancellorsville  (CG  62)  involving  an  ASCM 
surrogate  target  resulted  in  even  more  stringent  safety 
constraints. 
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-  In  addition  to  stand-off  ranges  (on  the  order  of  1.5  to 

5  nautical  miles  for  subsonic  and  supersonic  surrogates, 
respectively),  safety  restrictions  require  that  ASCM 
targets  not  be  flown  directly  at  a  manned  ship,  but  at  some 
cross- range  offset,  which  unacceptably  degrades  the 
operational  realism  of  the  test. 

-  Similar  range  safety  restrictions  will  preclude  manned  ship 
testing  of  eight  of  the  nine  ASCM  scenarios  contained  in 
the  Navy-approved  requirements  document  for  the  Aegis 
Modernization  Advanced  Capability  Build  16  Combat 
System  upgrade  as  well  as  testing  of  the  AMDR  minimum 
track  range  requirement  against  supersonic,  sea-skimming 
ASCM  threat-representative  surrogates  at  the  land-based 
AMDR  Pacific  Missile  Range  Facility  test  site. 

-  To  overcome  these  safety  restrictions  for  the  LHA-6, 
Littoral  Combat  Ship  (LCS),  DDG  1000,  LPD-17, 
LSD-41/49,  and  CVN-78  ship  classes,  the  Navy 
developed  an  Air  Warfare/Ship  Self  Defense  Enterprise 
modeling  and  simulation  (M&S)  test  bed  that  uses  live 
testing  in  the  close-in  battlespace  with  targets  flying 
realistic  threat  profiles  and  manned  ship  testing  for  other 
battlespace  regions  and  softkill  capabilities  to  validate 
and  accredit  the  M&S  test  bed.  The  same  needs  to  be 
done  for  the  DDG  5 1  Flight  III  Destroyer  with  its  AMDR. 
Side-by-side  comparison  between  credible  live  fire  test 
results  and  M&S  test  results  form  the  basis  for  the  M&S 
accreditation.  Without  an  SDTS  with  AMDR  and  an  Aegis 
Combat  System,  there  will  not  be  a  way  to  gather  all  of 
the  operationally  realistic  live  fire  test  data  needed  for 
comparison  to  accredit  the  M&S. 

•  The  Navy  needs  to  improve  its  Aegis  Weapon  System  (AWS) 

models  that  are  currently  provided  by  Lockheed  Martin’s 

Multi-Target  Effectiveness  Determined  under  Simulation  by 

Aegis  (MEDUSA)  M&S  tool. 

-  MEDUSA  encompasses  several  components  of  the  AWS 
including  the  SPY-1  radar,  Command  and  Decision, 
and  Weapon  Control  System.  MEDUSA  models  AWS 
performance  down  to  the  system  specification  and  the 
Navy  considers  it  a  high-fidelity  simulation  of  AWS. 

-  Flowever,  it  is  not  a  tactical  code  model,  so  its  fidelity 
is  ultimately  limited  to  how  closely  the  specification 
corresponds  to  the  Aegis  tactical  code  (i.e.,  the 
specification  is  how  the  system  is  supposed  to  work  while 
the  tactical  code  is  how  the  system  actually  works).  This 
adds  to  the  need  for  realistic  live  fire  shots  to  support 
validation  efforts. 

-  Earlier  test  events  highlight  the  limitations  of  specification 
models  like  MEDUSA.  During  Aegis  Advanced 
Capability  Build  08  testing  in  2011,  five  AWS  software 
errors  were  found  during  live  fire  events  and  tracking 
exercises.  Three  software  errors  contributed  to  a  failed 
SM-2  engagement,  one  to  a  failed  ESSM  engagement, 
and  one  to  several  failed  simulated  engagements  during 
tracking  exercises.  Since  these  problems  involved  software 
coding  errors,  it  is  unlikely  that  a  specification  model  like 
MEDUSA  (which  assumes  no  software  errors  in  tactical 


code)  would  account  for  such  problems  and  hence  it  would 
overestimate  the  combat  system’s  capability. 

-  By  comparison,  the  Air  Warfare/Ship  Self  Defense 
Enterprise  M&S  test  bed  used  for  assessing 

USS  S  an  Antonio’s  (LPD-17)  self-defense  capabilities  used 
re-hosted  Ship  Self-Defense  System  Mk  2  tactical  code. 

•  Since  Aegis  employs  ESSM  in  the  close-in,  self-defense 
battlespace,  understanding  ESSM’s  performance  is  critical 
to  understanding  the  self-defense  capabilities  of  the  DDG  51 
Flight  III  Destroyer. 

-  Past  DOT&E  annual  reports  have  stated  that  the  ESSM’s 
operational  effectiveness  has  not  been  determined.  The 
Navy  has  not  taken  action  to  adequately  test  the  ESSM’s 
operational  effectiveness. 

-  Specifically,  because  safety  limitations  preclude  ESSM 
firing  in  the  close-in  self-defense  battlespace,  there  are  very 
little  test  data  available  concerning  ESSM’s  performance, 
as  installed  on  Aegis  ships,  against  supersonic  ASCM 
surrogates. 

-  Any  data  available  regarding  ESSM’s  performance 
against  supersonic  ASCM  surrogates  are  from  a  Ship 
Self-Defense  System-based  combat  system  configuration, 
using  a  completely  different  guidance  mode  or  one  that  is 
supported  by  a  different  radar  suite. 

•  The  cost  of  building  and  operating  an  Aegis  SDTS  is 
small  when  compared  to  the  total  cost  of  the  AMDR 
development/procurement  and  the  eventual  cost  of  the 
22  (plus)  DDG  5 1  Flight  III  ships  that  are  planned  for 
acquisition  ($55+  Billion).  Even  smaller  is  the  cost  of  the 
SDTS  compared  to  the  cost  of  the  ships  that  the  DDG  51 
Flight  III  Destroyer  is  expected  to  protect  (approximately 
$450  Billion  in  new  ship  construction  over  the  next  30  years). 

-  If  DDG  51  Flight  III  Destroyers  are  unable  to  defend 
themselves,  these  other  ships  are  placed  at  substantial  risk. 

-  Moreover,  the  SDTS  is  not  a  one-time  investment  for  only 
the  AMDR/DDG  51  Flight  III  IOT&E,  as  it  would  be 
available  for  other  testing  that  cannot  be  conducted  with 
manned  ships  (e.g.,  the  ESSM  Block  2)  and  as  the  combat 
system  capabilities  are  improved. 

Recommendations 

•  Status  of  Previous  Recommendations.  There  are 

three  previous  recommendations  that  remain  valid.  The  Navy 
should: 

1 .  Program  and  fund  an  SDTS  equipped  with  the  AMDR  and 
DDG  5 1  Flight  III  Aegis  Combat  System  in  time  for  the 
DDG  51  Flight  III  Destroyer  IOT&E. 

2.  Modify  the  AMDR,  Aegis  Modernization,  and  DDG  5 1 
Flight  III  Test  and  Evaluation  Master  Plans  to  include  a 
phase  of  IOT&E  using  an  SDTS  equipped  with  the  AMDR 
and  DDG  51  Flight  III  Combat  System. 

3.  Modify  the  AMDR,  Aegis  Modernization,  and  DDG  5 1 
Flight  III  Test  and  Evaluation  Master  Plans  to  include  a 
credible  M&S  effort  that  will  enable  a  full  assessment 
of  the  AMDR  and  DDG  51  Flight  III  Combat  System’s 
self-defense  capabilities. 
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•  FY 14  Recommendation. 

1.  The  Navy  should  comply  with  the  DEPSECDEF  direction 
to  develop  and  fund  a  plan,  to  be  approved  by  DOT&E,  to 
conduct  at-sea  testing  of  the  self-defense  of  the  DDG  5 1 
Flight  111  Destroyer  with  the  AMDR  and  Aegis  Combat 
System. 
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Distributed  Common  Ground  System  -  Marine  Corps 

(DCGS-MC) 


Executive  Summary 

•  The  Marine  Corps  Operational  Test  and  Evaluation 
Activity  conducted  the  Distributed  Common  Ground 
System  -  Marine  Corps  (DCGS-MC)  IOT&E 

July  21  -  31,  2014,  at  Camp  Lejeune,  North  Carolina,  in 
accordance  with  the  DOT&E-approved  test  plan. 

•  During  IOT&E,  DCGS-MC  successfully  connected  to  the 
DCGS  Integrated  Backbone  (DIB),  allowing  Marines  to 
search  and  download  intelligence  from  joint  and  intelligence 
community  users.  However,  as  tested,  the  DCGS-MC 
Increment  1  did  not  enhance  the  user’s  ability  to  produce 
intelligence  products. 

•  The  Marine  Corps  Information  Assurance  Red  Team 
discovered  significant  cybersecurity  vulnerabilities  during  the 
IOT&E  that  require  correction  before  fielding.  Details  are 
provided  in  DOT&E’s  classified  IOT&E  report. 

•  The  DCGS-MC  did  not  meet  availability  or  reliability 
requirements  and  users  found  the  system  difficult  to  use. 

System 

•  DCGS-MC  is  an  Acquisition  Category  III  program. 

The  DCGS-MC  is  a  multi-level  secure,  integrated 
family-of-systems.  The  system  is  composed  of  a  new 
Enterprise  DIB  Service  node  that  delivers  web  application 
software  on  commercial  off-the-shelf  hardware,  integrated 
with  legacy  Tactical  Exploitation  Group  and  Topographic 
Production  Capability. 

•  The  DCGS-MC  provides  Marine  intelligence  analysts  access 
to  the  DIB.  The  DIB  provides  the  framework  that  allows 
sharing  of  intelligence  services  and  data  via  web  services. 

The  Army,  Navy,  Air  Force,  and  intelligence  agencies 
developed  and  fielded  their  own  versions  of  DCGS.  Via 

the  DIB,  intelligence  analysts  can  search  for  and  download 
intelligence  information  and  post  the  intelligence  product 
they  produce  for  others  to  use. 


Mission 

•  Marine  intelligence  analysts  will  use  the  DCGS-MC 
Enterprise  system  to  produce  geospatial  intelligence  products 
through  the  processing,  exploitation,  and  analysis  of  data 
derived  from  all  Marine  Corps  organic  intelligence  sources, 
nontraditional/battlefield  observation/collection  of  joint, 
multi-national  (coalition/allied)  partners  in  support  of  Marine 
Corps  operations,  and  tailored  theater  and  national  systems. 

•  The  Marine  Air  Ground  Task  Force  will  use  the  DCGS-MC  to 
connect  intelligence  professionals  with  multi-discipline  data 
sources,  analytic  assessments,  and  collection  assets  via  the  DIB. 

Major  Contractors 

•  Lead  System  Integrator:  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR)  Systems  Center 
Atlantic  -  Charleston,  South  Carolina 

•  SAIC  -  Charleston,  South  Carolina 


Activity 

•  The  DCGS-MC  Program  Office  completed  four  developmental 
tests  between  August  2012  and  November  2013,  followed  by 

a  period  of  fixes  and  subsequent  regression  testing  through 
May  2014.  The  Program  Office  used  the  developmental  test 
information  to  determine  that  the  system  was  ready  to  enter 
operational  test. 

•  The  Marine  Corps  Operational  Test  and  Evaluation 
Activity  conducted  the  IOT&E  from  July  21  -31,  2014, 
at  Camp  Lejeune,  North  Carolina,  in  accordance  with  the 


DOT&E-approved  test  plan.  Marine  Corps  intelligence 
analysts  answered  requests  for  information  using  operationally 
representative  systems  connected  to  networks  providing  access 
to  intelligence  information.  Analysts  used  the  DCGS-MC  to 
search  for  and  download  files  from  the  DIB,  modify  the  files, 
and  then  upload  the  modified  files  back  to  the  DIB. 

•  In  September  2014,  the  Marine  Corps  System  Command 
indefinitely  postponed  the  Full  Deployment  Decision. 
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Assessment 

•  During  the  IOT&E,  the  intelligence  analysts  using  the 
DCGS-MC  did  not  demonstrate  improved  mission  performance 
over  the  intelligence  analysts  using  the  legacy  systems.  The 
intelligence  analysts  using  the  DCGS-MC  answered  requests  for 
information  (RFIs)  with  the  same  quality  and  response  time  as 
the  intelligence  analysts  using  the  legacy  systems.  This  might 
be  partially  attributable  to  the  RFIs  not  requiring  extensive 

use  of  external  intelligence  information.  If  the  Marine  Corps 
implemented  updated  concept  of  operations  and  doctrine  to  take 
advantage  of  the  extensive  external  intelligence  available  via  the 
DIB,  DCGS-MC  might  show  more  operational  value. 

•  Marine  Corps  Information  Assurance  Red  Team  adversarial 
activities  and  on-site  cybersecurity  compliance  checks  and 
penetration  testing  identified  significant  cybersecurity  problems 
that  introduce  vulnerabilities  and  reduce  the  security  of  the 
system. 

•  DCGS-MC  did  not  satisfy  availability  and  reliability 
requirements,  but  did  satisfy  the  time  to  repair  requirement. 

-  The  web-portal  on  the  DIB  server  froze  regularly.  The 
root  cause  of  the  problem  was  not  discovered  during  the 
developmental  test. 

-  The  DCGS-MC  servers  were  not  synchronized  to  a  universal 
time  standard;  performance  was  degraded  when  drifting 
occurred. 


•  Marine  Corps  intelligence  analysts  considered  the  DCGS-MC 
to  be  more  difficult  to  use  than  the  current  systems.  Surveys 
revealed  that  they  generally  preferred  the  legacy  systems. 

•  On  October  31,  2014,  DOT&E  issued  a  classified  report  on 
the  DCGS-MC  IOT&E. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations.  The  Program  Office  should: 

1 .  Correct  the  cybersecurity  vulnerabilities  discovered  during 
IOT&E  and  verify  via  testing. 

2.  Execute  a  reliability  growth  program  with  testing  to 
confirm  improvement. 

3.  Provide  a  plan  to  implement  changes  to  improve  system 
usability  that  includes  verifying  improvements  using  the 
standard  System  Usability  Scale  survey. 

4.  Update  concept  of  operations  and  doctrine  to  take 
advantage  of  external  intelligence  information  available 
via  the  DIB. 
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E-2D  Advanced  Hawkeye 


Executive  Summary 

•  The  first  E-2D  FOT&E  period  (OT-D1)  started  in  2QFY 14 
to  evaluate  the  E-2D  Advanced  Hawkeye  Initial  Operational 
Capability  hardware/software  configuration.  Delta 
System/ Software  Configuration  (DSSC)  Build  1,  as  well 

as  carrier  suitability.  Testing  is  scheduled  to  complete  in 
1QFY15. 

•  The  E-2D  Advanced  Hawkeye  has  been  designated  an 
Acquisition  Category  1C  program.  On  August  8,  2014,  the 
Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test 
and  Evaluation  (DASD(DT&E))  approved  the  developmental 
test  and  evaluation  plan  within  the  E-2D  Test  and  Evaluation 
Master  Plan  (TEMP)  and  removed  the  E-2D  from  active 
DASD(DT&E)  oversight. 

•  The  E-2D  TEMP  revision  D  supports  the  FOT&E,  but 
does  not  state  requirements  or  resources  for  integrated  or 
operational  testing  of  Naval  Integrated  Fire  Control-Counter 
Air  (NIFC-CA)  From-The-Air  (FTA).  Change  1  to  revision 
D  is  expected  to  address  this  in  DSSC  Build  2.  Change  1  to 
revision  D  will  be  published  to  address  NIFC-CA  FTA  areas 
relevant  to  E-2D  only,  and  to  support  DSSC  Build  2  FOT&E 
in  FY16. 

•  Commander,  Operational  Test  and  Evaluation  Force  (COTF) 
conducted  a  Verification  of  Correction  of  Deficiencies  (VCD) 
assessing  the  program’s  progress  in  addressing  deficiencies 
found  during  IOT&E.  The  VCD  completed  in  1QFY14.  Not 
all  of  the  problems  identified  in  the  DOT&E  IOT&E  report 
were  resolved  during  the  VCD.  The  Navy  intends  to  address 
many  of  the  problems  through  a  series  of  hardware  and 
software  changes  that  will  be  incorporated  and  tested  through 
FY19. 

•  USD(AT&L)  approved  the  E-2D  for  a  multi-year  procurement 
contract  after  successful  VCD  completion  in  1QFY14. 

System 

•  The  E-2D  Advanced  Hawkeye  is  a  carrier-based  Airborne 
Early  Warning  and  Command  and  Control  aircraft. 

•  Significant  changes  to  this  variant  of  the  E-2  include  upgraded 
engines  to  provide  increased  electrical  power  and  cooling 
relative  to  current  E-2C  aircraft;  a  strengthened  fuselage  to 


support  increased  aircraft  weight;  replacement  of  the  radar 
system,  the  communications  suite,  and  the  mission  computer; 
and  the  incorporation  of  an  all-glass  cockpit,  which  permits 
the  co-pilot  to  act  as  a  tactical  fourth  operator  in  support  of  the 
system  operators  in  the  rear  of  the  aircraft. 

•  The  radar  upgrade  replaces  the  E-2C  mechanically-scanned 
radar  with  a  phased-array  radar  that  has  combined  mechanical 
and  electronic  scan  capabilities. 

•  The  upgraded  radar  provides  significant  improvement  in 
littoral  and  overland  detection  performance  and  Theater  Air 
and  Missile  Defense  capabilities. 

•  The  E-2D  Advanced  Hawkeye  System  includes  all  simulators, 
interactive  computer  media,  and  documentation  to  conduct 
maintenance,  as  well  as  aircrew  shore-based  initial  and 
follow-on  training. 

Mission 

The  Combatant  Commander,  whether  operating  from  the  aircraft 

carrier  or  from  land,  will  use  the  E-2D  Advanced  Hawkeye  to 

accomplish  the  following  missions: 

•  Theater  air  and  missile  sensing  and  early  warning 

•  Battlefield  management,  command,  and  control 

•  Acquisition,  tracking,  and  targeting  of  surface  warfare  contacts 

•  Surveillance  of  littoral  area  objectives  and  targets 

•  Tracking  of  strike  warfare  assets 

Major  Contractor 

Northrop  Grumman  Aerospace  Systems  -  Melbourne,  Florida 


Activity 

•  The  E-2D  Advanced  Hawkeye  has  been  designated  an 
Acquisition  Category  1C  program.  On  August  8,  2014, 
the  DASD(DT&E)  approved  the  developmental  test  and 
evaluation  plan  within  the  E-2D  TEMP  and  removed  the  E-2D 
from  active  DASD(DT&E)  oversight. 


•  In  4QFY13-1QFY14,  COTF  conducted  a  VCD  to  assess  the 
program’s  progress  in  addressing  deficiencies  found  during 
IOT&E  at  Naval  Air  Station  (NAS)  Patuxent  River,  Maryland; 
NAS  Fallon,  Nevada;  and  Point  Mugu,  California.  DOT&E 
reported  the  results  of  the  VCD  in  March  2014  to  inform  the 
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Defense  Acquisition  Board  decision  on  F  ull-Rate  Production 
Lot  2. 

•  USD(AT&L)  approved  the  E-2D  for  a  multi-year  procurement 
contract  after  successful  VCD  completion  in  1QFY14. 

•  In  3QFY14,  COTF  started  the  E-2D’s  first  FOT&E  period 
(OT-D1  phase)  to  assess  the  E-2D’s  operational  effectiveness 
and  suitability  to  support  the  Theater  Air  and  Missile  Defense 
mission.  Testing  also  evaluated  the  operational  effectiveness 
and  suitability  of  hardware  and  software  changes  incorporated 
in  DSSC  Build  1.  The  Navy  conducted  testing  at  NAS 
Patuxent  River,  Maryland;  Flolloman  AFB,  New  Mexico; 
White  Sands  Missile  Range  (WSMR),  New  Mexico;  and 
Point  Mugu,  California.  FOT&E  will  complete  in  FY15. 

•  Limited  E-2D  carrier  testing  was  conducted  in  IOT&E. 

Carrier  testing  to  determine  carrier  suitability  was  conducted 
as  part  of  FOT&E  in  September  and  October  2014. 

•  Not  all  of  the  problems  identified  in  the  DOT&E  IOT&E 
report  were  resolved  during  the  VCD.  The  Navy  intends  to 
address  many  of  these  problems  through  a  series  of  hardware 
and  software  changes  that  will  be  incorporated  and  tested 
through  FY19. 

•  The  Navy  continued  to  correct  deficiencies  with 
E-2D-Cooperative  Engagement  Capability  performance  with 
a  plan  to  have  deficiencies  remedied  in  FY17  with  fielding  of 
DSSC  Build  3. 


Assessment 

•  DOT&E  provided  a  classified  assessment  of  the  VCD  results 
in  March  2014.  Testing  was  limited  and  the  data  collected 
were  insufficient  to  resolve  major  problems  stated  in  the 
classified  February  2013  E-2D  Advanced  Hawkeye  IOT&E 
report.  However,  the  VCD  data  did  provide  information 

on  training  not  assessed  during  IOT&E.  Complete  training 
on  all  parts  of  the  E-2D  system  was  not  satisfactory.  A  full 
assessment  of  E-2D  operational  capabilities  will  require  future 
operational  tests  and  systematic  updates. 

•  The  E-2D  TEMP  revision  D  supports  the  FOT&E,  but  does  not 
state  requirements  or  resources  for  integrated  or  operational 
testing  ofNIFC-CAFTA.  Change  1  to  revision  D  is  expected 
to  address  this  in  DSSC  Build  2. 

•  FOT&E  testing  is  not  complete.  DOT&E  is  currently 
observing  test  events  and  analyzing  data  from  the  testing. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  continues 
efforts  to  improve  radar  and  mission  system  performance,  and 
radar  and  overall  weapon  system  reliability  and  availability  as 
previously  recommended. 

•  FY 14  Recommendation. 

1.  The  Navy  should  provide  complete  training  on  all 
components  of  the  E-2D  system  and  missions. 


178 


E-2D  Hawkeye 


F Y1  4  NAVY  PROGRAMS 


F/A-18E/F  Super  Hornet  and  EA-18G  Growler 


Executive  Summary 

•  From  FY12  into  FY14,  the  Navy  conducted  Software 
Qualification  Testing  (SQT)  of  System  Configuration  Set 
(SCS)  H8E  for  the  F/A-18E/F  Super  Hornet  and  EA-18G 
Growler  aircraft.  The  testing  was  conducted  in  two  phases. 
The  first  phase  tested  software  improvements  in  SCS  H8E, 
while  the  second  phase  tested  improvements  in  the  APG-79 
Active  Electronically  Scanned  Array  (AESA)  radar.  DOT&E 
intends  to  issue  a  single  report  covering  both  phases  of  SCS 
H8E  testing  in  FY15. 

•  Although  the  reliability  of  the  APG-79  radar  improved  in 
both  phases  of  testing,  problems  with  software  stability  in  the 
AESA  radar  resulted  in  failure  to  meet  reliability  and  built-in 
test  (BIT)  performance  requirements. 

•  The  F/A-18E/F  Super  Hornet  weapon  system  continues  to 
demonstrate  operational  effectiveness  and  suitability  for 
most  threat  environments;  however,  the  platform  is  not 
operationally  effective  in  specific  threat  environments,  which 
are  detailed  in  previous  DOT&E  classified  reports. 

•  The  EA-18G  Growler  weapons  system  equipped  with  SCS 
H8E  is  operationally  suitable  and  operationally  effective  with 
the  same  radar  limitations  as  the  E/F. 

•  The  Navy  began  SQT  of  SCS  25X  for  early-model  F/A-18 
aircraft  in  July  2014. 

System 

F/A-18E/F  Super  Hornet 

•  The  Super  Hornet  is  the  Navy’s  premier  strike -fighter 
aircraft  and  is  a  more  capable  follow-on  replacement  to  the 
F/A-18A/B/C/D  and  the  F-14. 

•  F/A-18E/F  Lot  25+  aircraft  provide  functionality  essential 
for  integrating  all  Super  Hornet  Block  2  hardware  upgrades, 
which  include: 

-  Single  pass  multiple  targeting  for  GPS-guided  weapons 

-  Use  of  off-board  target  designation 

-  Improved  datalink  for  target  coordination  precision 

-  Implementation  of  air-to-ground  target  aim  points 

•  Additional  systems  include: 

-  APG-73  or  APG-79  radar 

-  Advanced  Targeting  Forward-Looking  Infrared  System 

-  AIM-9  infrared-guided  missiles  and  AIM- 120  and 
AIM-7  radar-guided  missiles 

-  Multi-functional  Information  Distribution  System  for 
Link  16  tactical  datalink  connectivity 

-  Joint  Helmet-Mounted  Cueing  System 

-  Integrated  Defensive  Electronic  Countermeasures 
EA-18G  Growler 

•  The  Growler  is  the  Navy’s  land-  and  carrier-based,  radar 
and  communication  jamming  aircraft. 


•  The  two-seat  EA-18G  replaces  the  four-seat  EA-6B.  The 
new  ALQ-218  receiver,  improved  connectivity,  and  linked 
displays  are  the  primary  design  features  implemented  to 
reduce  the  operator  workload  in  support  of  the  EA-18G’s 
two-person  crew. 

•  The  Airborne  Electronic  Attack  (AEA)  system  includes: 

-  Modified  E A-6B  Improved  Capability  III  ALQ-2 1 8 
receiver  system 

-  Advanced  crew  station 

-  Legacy  ALQ-99  j  amming  pods 

-  Communication  Countermeasures  Set  System 

-  Expanded  digital  Link  1 6  communications  network 

-  Electronic  Attack  Unit 

-  Interference  Cancellation  System  that  supports 
communications  while  jamming 

-  Satellite  receive  capability  via  the  Multi-mission 
Advanced  Tactical  Terminal 

•  Additional  systems  include: 

-  APG-79  AESA  radar 

-  Joint  Helmet-Mounted  Cueing  System 

-  High-speed  Anti-Radiation  Missile 

-  AIM- 120  radar-guided  missiles 
System  Configuration  Set  (SCS)  Software 

•  Growler  and  Super  Hornet  aircraft  employ  SCS  operational 
software  to  enable  major  combat  capabilities.  All  EA-18Gs 
and  Block  2  F/A-18s  (production  Lot  25  and  beyond) 

use  high-order  language  or  “H-series”  software,  while 
F/A-18E/F  prior  to  Lot  25  and  all  legacy  F/A-18  A/B/C/D 
aircraft  use  “X-series”  software. 

-  The  Navy  released  H8E  Phase  I  this  year  and  is  currently 
flying  23X  in  earlier  lot  aircraft. 

-  H8E  Phase  II  (H-series)  has  recently  completed  test  and 
25X  (X-series)  is  currently  under  test. 
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Mission 

•  Combatant  Commanders  use  the  F/A-18E/F  to: 

-  Conduct  offensive  and  defensive  air  combat  missions 

-  Attack  ground  targets  with  most  of  the  U.S.  inventory  of 
precision  and  non-precision  weapon  stores 

-  Provide  in-flight  reflieling  for  other  tactical  naval  aircraft 

-  Provide  the  fleet  with  an  organic  tactical  reconnaissance 
capability 

•  Combatant  Commanders  use  the  EA-18G  to: 

-  Support  friendly  air,  ground,  and  sea  operations  by 
countering  enemy  radar  and  communications 

-  Jam  integrated  air  defense  systems 

-  Support  non-integrated  air  defense  missions  and  emerging 
non-lethal  target  sets 


-  Enhance  crew  situational  awareness  and  mission 
management 

-  Enhance  connectivity  to  national,  theater,  and  tactical  strike 

assets 

-  Provide  enhanced  lethal  suppression  through  accurate 
High-speed  Anti-Radiation  Missile  targeting 

-  Provide  the  EA-18G  crew  air-to-air  self-protection  with  the 
AIM- 120 

Major  Contractor 

The  Boeing  Company,  Integrated  Defense 
Systems  -  St.  Louis,  Missouri 


Activity 

•  The  Navy  conducted  F/A- 1 8E/F  and  EA- 1 8G  H8E  SQT 
in  two  phases.  Phase  I  testing,  which  focused  on  system 
software  improvements,  occurred  between  June  2012  and 
May  2013.  Test  aircraft  equipped  with  SCS  H8E  accumulated 
1,296.0  flight  hours. 

•  SCS  H8E  Phase  II  testing  focused  on  APG-79  improvements. 
Super  Hornet  aircraft  equipped  with  the  APG-79  radar 
accumulated  1,884.4  flight  hours  during  Phase  II  testing, 
which  took  place  between  October  2013  and  June  2014. 

•  The  Navy  began  testing  SCS  25X  in  July  2014,  for  use  on 
earlier  model  F/A- 18  aircraft,  with  completion  expected  in 
FY15.  Developmental  delays  pushed  the  start  of  this  test  back 
for  a  year  from  the  original  schedule. 

•  The  Navy  has  continued  to  defer  development  of  the  AESA’s 
full  electronic  warfare  capability  to  later  software  builds  and 
plans  to  test  this  capability  in  SCS  H12. 

Assessment 

•  SCS  H8E  demonstrated  incremental  improvements  in 
capability  in  Phase  I.  APG-79  reliability  improved  during 
both  Phases  1  and  II  testing  compared  to  previous  operational 
tests  and  provides  improved  performance  compared  to  the 
legacy  APG-73  radar  employed  on  earlier  F/A- 18  aircraft. 
Nonetheless,  key  deficiencies  in  operational  performance 
remain  to  be  addressed. 

•  While  the  AESA  radar  demonstrated  improved  reliability, 
radar  software  instability  resulted  in  failure  to  meet  reliability 
and  BIT  performance  requirements. 

•  The  Navy  has  begun  to  address  long-standing  deficiencies 
in  air  warfare  during  H8E.  The  F/A-18E/F  Super  Hornet 


weapons  system  is  operationally  effective  and  suitable  for 
some  threat  environments.  However,  as  noted  in  previous 
DOT&E  classified  reports,  there  are  current,  more  stressing 
threat  environments  in  which  the  F/A- 18  remains  not 
operationally  effective. 

•  The  EA-18G  Growler  remains  operationally  effective  and 
suitable  subject  to  the  same  threat  environment  limitations 
as  the  E/F.  SCS  H8E  testing  indicates  that  geolocation 
accuracy  and  timeliness  has  improved  since  the  H6E  testing, 
but  timeliness  with  the  jammers  off  still  has  room  for 
improvement. 

•  SCS  H8E  testing  did  not  include  an  end-to-end  multi- AIM- 120 
missile  shot.  This  Navy  operational  capability  has  not  been 
demonstrated  previously  in  a  successful  test.  The  Navy 
tentatively  plans  to  conduct  a  multi-missile  shot  as  part  of  SCS 
H12  testing. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  made 
progress  on  addressing  FY13  recommendations  to  continue  to 
improve  the  reliability  and  BIT  functionality  of  the  APG-79 
radar,  but  that  recommendation  remains  valid.  Additionally, 
recommendations  to  conduct  an  operationally  representative 
end-to-end  missile  test  to  demonstrate  APG-79  radar  and 
system  software  support  for  a  multiple  AIM- 120  missile 
engagement  and  to  develop  and  characterize  the  full  electronic 
warfare  capability  of  the  APG-79  radar  remain  unchanged. 

•  FY 14  Recommendations.  None. 
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Global  Command  and  Control  System  -  Maritime 

(GCCS-M) 


Executive  Summary 

•  The  Navy’s  Commander,  Operational  Test  and  Evaluation 
Force  (COTF)  conducted  the  IOT&E  of  the  Global  Command 
and  Control  System  -  Maritime  (GCCS-M)  version  (v4. 1) 
Group  Level  variant  onboard  the  USS  Milius  from 

May  28  through  June  13,  2014.  GCCS-M  v4.1  occasionally 
experiences  a  low  memory  condition,  making  the  system  run 
too  slowly  to  be  operationally  effective.  Users  performed 
a  preemptive  12-minute  reboot  of  the  GCCS-M  v4.1  client 
prior  to  daily  operational  exercise  participation  to  prevent  this 
condition.  DOT&E  determined  that  the  Group  Level  variant 
is  operationally  effective  (with  the  operational  workaround  for 
the  low  memory  condition)  and  operationally  suitable. 

•  The  Navy  Information  Operations  Command  (NIOC) 

Red  Team  attempted  to  penetrate  and  exploit  the  GCCS-M 
v4.1  system  during  the  IOT&E  and  identified  two  major 
and  two  minor  cybersecurity  deficiencies.  The  two  major 
deficiencies  were  attributed  to  the  Consolidated  Afloat 
Networks  and  Enterprise  Services  (CANES)  program,  which 
interfaces  with  GCCS-M  v4. 1 .  The  two  minor  deficiencies 
were  a  residt  of  inadequate  system  documentation  and  a 
non-responsive  helpdesk,  which  prevented  the  user  from 
restoring  the  system.  DOT&E  determined  that  the  Group 
Level  variant  is  not  survivable  until  cybersecurity  deficiencies 
have  been  corrected. 

System 

•  GCCS-M  is  a  command,  control,  communications,  computers, 
and  intelligence  system  consisting  of  software,  procedures, 
standards,  and  interfaces  that  provide  an  integrated  near 
real-time  picture  of  the  battlespace  used  to  conduct  joint  and 
multi-national  maritime  operations. 

•  The  Navy  is  developing  GCCS-M  Increment  2  at  the  Force, 
Group,  and  Unit  Levels.  Force  Level  includes  aircraft  carrier 
(CVN),  amphibious  assault  (LHA  and/or  LHD),  and  command 
ships  (LCC).  Group  Level  includes  guided  missile  cruisers 
(CG),  destroyers  (DDG),  and  submarines.  Unit  Level  includes 
guided  missile  frigates,  dock  landing  ships,  amphibious 
transport  docks,  and  patrol  coastal  craft. 

•  GCCS-M  Increment  2  consists  of  two  distinct  types  of 
software: 

-  Aircraft  carrier,  amphibious  command  ship  (LCC), 
and  amphibious  assault  ship  capability  based  on  the 
GCCS- Joint  software  baseline 


-  Guided  missile  cruiser  and  below  capability  based  on  the 
extensible  Common  Operational  Picture  software  baseline 

•  The  Navy  intends  to  release  the  Group  and  Unit  Level  solution 
in  a  three  configuration  phased  approach,  starting  with  the 
patrol  coastal  ships,  then  the  full  Unit  Level  ships,  and  finally 
the  Group  Level  ships. 

Mission 

•  U.S.  maritime  commanders  utilize  GCCS-M  to  exercise 
command  and  control  over  forces  in  support  of  maritime 
operations. 

•  Commanders  at  all  echelons  use  GCCS-M  to: 

-  Integrate  scalable  command  and  control,  communications, 
and  intelligence  capabilities 

-  Support  the  decision-making  process 

-  Process,  correlate,  and  display  geographic  track 
information  on  friendly,  hostile,  and  neutral  land,  sea,  air, 
and  space  forces,  integrated  with  available  intelligence  and 
environmental  information 

Major  Contractor 

Northrop  Grumman  Mission  Systems  -  San  Diego,  California 
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Activity 

•  COTF  conducted  the  IOT&E  of  GCCS-M  v4. 1  Group 
Level  variant  onboard  the  USS  Milius  from  May  28  through 
June  13,  2014,  in  accordance  with  the  DOT&E-approved 
IOT&E  Plan. 

-  The  USS  Milius  was  pier-side  at  Naval  Base 

San  Diego,  California,  from  May  28  through  June  8,  2014, 
and  underway  in  the  Southern  California  Operations  Area 
for  the  remainder  of  the  test  period. 

-  Combat  Information  Center  Battle  Force  Team  Trainer  and 
Surface-Launched  Attack  Missile  Exercise  scenarios  were 
executed  to  stimulate  the  GCCS-M  v4. 1  system. 

•  Concurrently  with  IOT&E,  a  NIOC  Red  Team  performed 
a  cybersecurity  assessment  that  included  system  scans, 
penetration  testing,  and  malicious  insider  analysis. 

•  DOT&E  intends  to  publish  an  IOT&E  report  in  FY15. 

Assessment 

•  The  Group  Level  variant  of  the  GCCS-M  v4. 1  system  is 
operationally  effective,  with  the  operational  workaround  for 
the  low  memory  condition.  GCCS-M  v4.1  performed  its 
command,  control,  and  communications  mission  well  during 
IOT&E. 

-  USS  Milius  maintained  an  integrated  common  operating 
picture  with  the  Commander,  Third  Fleet  Maritime 
Operations  Center  (MOC)  successfully  exchanging, 
processing,  and  displaying  near  real-time  track  data. 

-  Users  successfully  created,  displayed,  and  stored  overlay 
tactical  decision  aids  in  the  Common  Tactical  Picture 
Manager  and  exchanged  them  with  the  Third  Fleet  MOC. 

-  GCCS-M  v4. 1  received  and  displayed  historical  and 
current  Blue  Force  Tracking  data  from  the  Joint  Tactical 
Terminal. 

-  GCCS-M  v4. 1  received  position  and  intended  movement 
data  over  the  Navigation  Sensor  System  Interface 
(NAVSSI). 

•  Testing  identified  four  defects  affecting  operational 
effectiveness. 

-  First,  GCCS-M  v4.1  occasionally  experiences  a  low 
memory  condition,  making  the  system  run  too  slowly  to 
be  operationally  effective.  Users  performed  a  preemptive 
12-minute  reboot  of  the  GCCS-M  v4. 1  client  prior  to  daily 
operational  exercise  participation  to  prevent  this  condition. 

-  Second,  during  Undersea  Warfare  (USW)  operations, 
GCCS-M  v4. 1  did  not  transmit  tracks  and  overlay 
messages  to  the  Tactical  Decision  Support  System  (TDSS). 
The  mission  impact  was  minor  because  TDSS  was  the 
primary  decision  aid  for  USW  operations  and  relied  on 
the  USW  Combat  System  as  the  primary  source  for  track 
messages. 

-  Third,  10  surface  training  tracks  out  of  80  randomly 
displayed  incorrect  courses  and  speeds;  5  of  the  10  affected 
tracks  stopped  displaying.  Users  were  able  to  reintroduce 
tracks  into  the  common  operating  picture  to  accomplish 


training  objectives.  This  defect  was  isolated  to  local 
training  tracks,  as  operational  tracks  were  not  affected. 

-  Fourth,  GCCS-M  v4. 1  sometimes  starts  up  in  a  condition 
where  it  is  unable  to  accept  own  ship  position  information 
from  the  NAVSSI.  Since  this  condition  occurs  randomly 
and  infrequently,  the  system  is  recoverable  by  performing  a 
12-minute  reboot. 

•  The  Group  Level  variant  of  the  GCCS-M  v4. 1  system 
is  operationally  suitable.  GCCS-M  v4.1  met  reliability, 
maintainability,  and  availability  requirements. 

-  The  GCCS-M  v4. 1  Mean  Time  Between  Operational 
Mission  Failure  was  372  hours  (100  hours  or  greater 
required). 

-  GCCS-M  v4. 1  corrective  maintenance  for  one  operational 
mission  failure  was  12  minutes  (40  minutes  or  less 
required). 

-  Operational  availability  was  0.99  (0.95  or  greater 
required). 

-  GCCS-M  v4. 1  training  provides  the  knowledge  necessary 
to  operate  GCCS-M  in  its  typical  missions,  when 
supplemented  with  on-the-job  training.  Program  Office 
processes  and  procedures  used  to  identify,  track,  and 
correct  system  defects  were  effectively  used  to  support  and 
sustain  the  system. 

•  DOT&E  determined  that  the  Group  Level  variant  is  not 
survivable  until  cybersecurity  deficiencies  have  been 
corrected.  NIOC  identified  two  major  and  two  minor 
cybersecurity  deficiencies  during  IOT&E.  The  two  major 
deficiencies  were  attributed  to  the  CANES  program,  which 
interfaces  with  GCCS-M  v4. 1 .  The  two  minor  deficiencies 
were  a  result  of  inadequate  system  documentation  and  a 
non-responsive  helpdesk,  which  prevented  the  user  from 
restoring  the  system. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  addressed  all 
previous  recommendations. 

•  FY 14  Recommendations.  The  Navy  should: 

1.  Correct  the  following  GCCS-M  v4. 1  Group  Level 
deficiencies — 

■  Occasional  low  memory  condition  requiring  daily  reboot 

■  Lack  of  tracks  and  overlay  messages  to  the  TDSS 

■  Displaying  incorrect  surface  training  track  information 

■  Unable  to  accept  own  ship  position  information  from  the 
NAVSSI  upon  startup 

■  Two  cybersecurity  deficiencies  relating  to  system 
restoration 

2.  Review  CANES-equipped  ships  to  ensure  the  two  major 
cybersecurity  deficiencies  are  corrected. 

3.  Implement  an  operational  workaround  to  periodically 
reboot  the  GCCS-M  v4. 1  client  to  prevent  a  low  memory 
condition,  until  the  defect  can  be  corrected. 
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Infrared  Search  and  Track  (IRST) 


Executive  Summary 

•  The  first  flight  of  the  Engineering  Development  Model 
(EDM)  Infrared  Search  and  Track  (IRST)  system  pod  on  the 
F/A-18E/F  took  place  on  February  12,  2014. 

•  The  Commander,  Operational  Test  and  Evaluation  Force 
(COTF)  conducted  an  operational  assessment  (OA)  to  inform 
a  December  2,  2014,  Milestone  C  decision. 

•  DOT&E  reported  on  the  OA  results  in  a  classified 
report  to  the  Milestone  Decision  Authority,  the  Principal 
Military  Deputy  to  the  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition  (ASN(RDA)) 
in  December  2014.  DOT&E  found  that  the  system  was 
progressing  towards  meeting  reliability  requirements.  The 
system  did  not  demonstrate  tracking  performance  needed  for 
weapons  employment. 

•  The  Milestone  C  decision  will  consider  approval  of  14  Block  I 
low-rate  initial  production  units  and  a  plan  to  transition  to 
Acquisition  Category  (ACAT)  I  status. 

System 

•  The  IRST  system  consists  of  a  passive  long-wave  infrared 
receiver  (IRR),  a  processor,  inertial  measurement,  and 
environmental  control  unit. 

•  The  Navy  designed  the  IRST  to  be  flown  on  the  F/A-18E/F 
and  it  will  be  built  into  a  modified  centerline  fuel  tank. 

•  The  Navy  is  developing  and  fielding  the  system  in  two  blocks: 
Block  1  will  reach  Initial  Operational  Capability  (IOC)  in 
FY18  and  use  components  from  the  F-15K/SG  IRR  that 
derive  from  the  F-14  IRST  system.  Block  II  will  begin  after 
the  Block  I  Full-Rate  Production  Decision  Review  and  will 
include  an  improved  IRR  and  updated  processors. 

•  The  Navy  intends  to  produce  a  total  of  170  IRST  systems. 
There  will  be  60  Block  I  sensors,  which  will  eventually  be 


updated  to  the  Block  II  configuration;  the  Navy  will  build  an 
additional  110  Block  II  sensors. 

Mission 

Commanders  will  use  IRST  in  a  radar-denied  environment  to 
locate  and  destroy  enemy  forces.  The  IRST  system  is  intended 
to  allow  the  F/A-18E/F  to  operate  and  survive  against  existing 
and  emerging  air  threats  by  enhancing  situational  awareness  and 
providing  the  ability  to  acquire  and  engage  targets  beyond  visual 
range. 

Major  Contractors 

•  The  Boeing  Company  -  St  Louis,  Missouri 

•  Lockheed  Martin  -  Orlando,  Florida 


Activity 

•  COTF  conducted  an  OA  in  two  phases  in  accordance  with 
DOT&E-approved  test  plans. 

•  Phase  I  of  the  OA  consisted  of  contractor  flight  testing  on  a 
flying  test  bed  between  July  and  October  2013  and  events 
flown  by  operational  test  pilots  October  29  -  31,  2013,  in 
Boeing’s  manned  simulator  facilities. 

•  The  first  flight  of  the  EDM  IRST  pod  on  the  F/A-18E/F  took 
place  on  February  12,  2014. 

•  Phase  II  of  the  OA  included  a  maintenance  demonstration  on 
April  2,  2014,  and  six  flights  on  IRST-equipped  F/A-18E/F 
developmental  test  aircraft  between  May  14  and  July  16,  2014. 
The  OA  served  both  developmental  and  operational  test 
objectives. 


•  DOT&E  approved  the  Milestone  C  Test  and  Evaluation  Master 
Plan  (TEMP)  in  October  2014. 

•  In  December  2014,  DOT&E  reported  on  the  OA  results  in 
a  classified  report  to  the  Milestone  Decision  Authority,  the 
Principal  Military  Deputy  to  the  ASN(RDA). 

Assessment 

•  IRST,  as  tested  in  the  OA,  demonstrated  unsatisfactory 
tracking  performance.  The  individual  problems  seen  vary  in 
the  degree  to  which  they  can  be  addressed  through  incremental 
improvements  and  are  reflective  of  immature  system  design. 
Present  track  quality  represents  a  significant  risk  that  the 
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system  to  be  tested  in  IOT&E  may  not  be  effective.  No 
decision  has  been  made  whether  entry  into  IOT&E  will  be 
delayed  in  order  to  fix  these  problems.  Further  details  can  be 
found  in  DOT&E’s  classified  report  to  the  Navy  acquisition 
executive. 

•  Prior  to  the  OA,  the  F/A-18E/F  mission  computer’s 
multi-sensor  integration  (MSI)  software  had  difficulty 
integrating  IRST  data  with  data  from  other  sensors  (e.g., 
radar).  This  is  to  be  expected  given  the  poor  quality  of 
information  provided  by  IRST;  however,  how  MSI  will 
perform  with  better  information  from  IRST  is  unknown.  As 
a  result,  COTF  did  not  test  MSI  during  the  OA.  Historically, 
integration  of  information  from  disparate  sensors  has  been 
difficult;  thus,  IRST  MSI  will  require  careful  developmental 
testing  and  analysis  to  ensure  the  successful  integration  of 
IRST  track  data. 

•  The  Key  Performance  Parameter  (KPP)  for  detection  and 
tracking  does  not  adequately  represent  the  operational 
environment  where  the  Navy  will  employ  IRST.  Presently,  the 
KPP  can  only  be  met  in  an  environment  that  does  not  reflect 
realistic  fighter  employment  and  tactics. 

•  Although  data  are  limited,  reliability  and  maintainability  as 
demonstrated  during  the  OA  flights  and  prior  developmental 
testing  are  consistent  with  a  system  with  a  rigorous  reliability 
growth  program. 

•  No  hardware  failures  were  experienced  during  flight  test 
on  F/A-18E/F  prior  to  or  during  the  OA.  While  software 
reliability  requires  significant  improvement  prior  to  IOT&E, 
fixes  have  been  identified  for  the  problems  seen  and  the 
Program  Office  has  a  process  in  place  to  track  and  correct 
errors  as  they  are  encountered.  Most  software  reliability 


problems  were  related  to  Built-in  Test  (BIT),  which  exhibited 
a  high  frequency  of  false  alarms.  The  most  serious  issue  was 
tracker  process  failures  not  detected  by  BIT.  The  program  has 
identified  a  fix  for  this  problem  and  is  currently  in  flight  test 
for  evaluation. 

•  A  maintenance  demonstration  indicates  removal  and 
replacement  times  should  be  adequate  to  meet  requirements. 
The  demonstration  identified  improvements  to  technical 
documentation  and  system  design  that  should  be  incorporated. 

•  Line-of-sight  estimation  algorithms  (known  as  servo-transfer 
alignment)  are  not  working  and  have  required  an  alternate 
approach  for  boresighting  the  system.  A  mechanical  boresight 
procedure  was  used  for  the  OA  and  might  serve  as  a  permanent 
solution  if  the  servo-transfer  alignment  algorithm  cannot  be 
improved.  An  assessment  is  being  made  to  determine  what  the 
logistical  impact  would  be.  The  assessment  will  examine  how 
long  the  manual  boresight  is  expected  to  hold  and  whether  new 
support  equipment  will  be  required  to  re -boresight  the  system 
aboard  ship. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Explicitly  state  detection  and  tracking  requirements  for  the 
range  of  operational  conditions  in  which  the  Navy  expects 
to  employ  the  system. 

2.  Improve  detection  and  tracking  performance  prior  to  entry 
into  IOT&E. 
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Integrated  Defensive  Electronic  Countermeasures 

(IDECM) 


Executive  Summary 

•  The  Navy  began  the  Integrated  Defensive  Electronic 
Countermeasures  (IDECM)  Block  IV  integrated  tests  in 
July  2013  and  completed  the  planned  testing  in  April  2014. 

-  The  integrated  tests  included  laboratory  testing  at  the 
Navy’s  Electronic  Combat  System  Evaluation  Laboratory, 
Point  Mugu,  California,  against  two  classified  threats,  as 
well  as  a  dense  emitter  scenario  and  flight  testing  at  the 
Electronic  Combat  Range  (ECR),  China  Lake  Naval  Air 
Station,  California,  and  the  Nevada  Test  and  Training 
Range  (NTTR). 

•  The  Navy  has  convened  an  F/A-18  wingman  compatibility 
working  group  that  has  members  from  multiple  Navy  program 
offices  to  investigate  and  resolve  deficiencies  associated  with 
the  aircraft  radar,  which  may  be  caused  by  other  sub-systems 
such  as  IDECM. 

•  The  Navy  accomplished  an  Operational  Test  Readiness 
Review  (OTRR)  for  IDECM  Block  IV  FOT&E  on 
June  16,  2014.  DOT&E  concurred  with  the  Navy  in 
commencing  operational  testing,  but  DOT&E  had  three  points 
of  concern: 

-  The  ambitious  schedule  for  completing  flight  test  at  NTTR 
in  only  two  range  periods 

-  Only  two  F/A-18C/D  aircraft  being  available  for  tests, 
which  could  create  schedule  delays  if  F/A-18C/D 
maintenance  problems  occurred 

-  The  potential  the  schedule  would  need  to  be  extended 
since  threat  hardware  at  both  ECR  and  NTTR  had  been 
experiencing  failures 

•  FOT&E  began  in  June  2014  and  is  scheduled  to  end  in 
February  2015.  DOT&E  will  produce  a  classified  FOT&E 
report  assessing  system  operational  effectiveness  and 
suitability  after  the  conclusion  of  the  IDECM  Block  IV 
FOT&E. 

System 

•  The  IDECM  system  is  a  radio  frequency,  self-protection 
electronic  countermeasure  suite  on  F/A-18  aircraft.  The 
system  is  comprised  of  onboard  and  off-board  components. 
The  onboard  components  receive  and  process  radar 
signals  and  can  employ  onboard  and/or  off-board  jamming 
components  in  response  to  identified  threats. 

•  There  are  four  IDECM  variants:  Block  I  (IB- 1),  Block  II 

(IB  2),  Block  III  (IB-3),  and  Block  IV  (IB-4).  All  four  variants 
include  an  onboard  radio  frequency  receiver  and  jammer. 

-  IB-1  (fielded  FY02)  combined  the  legacy  onboard 
receiver/jammer  (ALQ-165)  with  the  legacy  (ALE-50) 
off-board  towed  decoy. 


-  IB-2  (fielded  FY04)  combined  an  improved  onboard 
receiver/jammer  (ALQ-214)  with  the  legacy  (ALE-50) 
off-board  towed  decoy. 

-  IB-3  (fielded  FY11)  combined  the  improved  onboard 
receiver/jammer  (ALQ-214)  with  a  new  (ALE-55) 
off-board  fiber-optic  towed  decoy  that  is  more  integrated 
with  the  ALQ-214. 

-  IB-4  (currently  in  test)  is  intended  to  replace  the  onboard 
receiver/jammer  (ALQ-214(V)3)  with  a  lightweight, 
repackaged  onboard  jammer  (ALQ-214(V)4  and 
ALQ-214(V)5).  IB -4  also  replaces  the  ALQ-126B 

to  provide  advanced,  carrier-capable  jamming  to  the 
F/A-  18C/D  for  the  first  time. 

•  An  additional  program  to  provide  IDECM  Block  IV  the 
capability  to  deny  or  delay  targeting  of  the  F/A-18  by  enemy 
radars,  known  as  the  Software  Improvement  Program,  is  in 
early  development. 

•  The  F/A-18E/F  installation  includes  off-board  towed  decoys. 
The  F/A-18C/D  installation  includes  only  the  onboard 
receiver/jammer  components  and  not  the  towed  decoy. 

Mission 

•  Combatant  Commanders  will  use  IDECM  to  improve  the 
survivability  of  Navy  F/A-18  strike  aircraft  against  radio 
frequency-guided  threats  while  flying  air-to-air  and  air  to 
ground  missions. 

•  The  Navy  intends  to  use  IB-3’s  and  lB-4’s  complex  jamming 
capabilities  to  increase  survivability  against  modern 

radar- guided  threats. 

Major  Contractors 

•  ALE-55:  BAE  Systems  -  Nashua,  New  Hampshire 

•  ALQ-214:  Exelis,  Inc.  -  Clifton,  New  Jersey 

•  ALE-50:  Raytheon  Electronic  Warfare 
Systems  -  Goleta,  California 
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Activity 

IDECM  Block  III 

•  DOT&E  completed  its  IDECM  Block  III  IOT&E  report  in 
June  2011,  assessing  the  system  as  operationally  effective 
and  suitable  for  combat.  The  Navy  authorized  IDECM 
Block  III  full-rate  production  in  July  2011. 

•  The  Navy  held  In  Process  Review  #3  in  March  2012  and 
decided  to  begin  implementing  the  IDECM  Block  IV 
change  proposal  so  that  in  April  2012,  the  IDECM  Block  III 
production  line  transitioned  to  producing  IDECM  Block  IV 
systems  (production  lot  buy  9). 

IDECM  Block  IV 

•  The  Navy  held  In  Process  Review  #5  in  January  2014 
to  determine  if  the  IDECM  Block  IV  system  should  be 
approved  for  production  lot  buy  1 1  (the  program  is  in 
full-rate  production  and  In  Process  Reviews  support  lot 
buy  decisions).  The  Navy  approved  the  FY 14  production 
decision  and  directed  the  Program  Office  to  accomplish 
another  In  Process  Review  (#6)  to  support  the  next 
production  lot  buy  decision  after  FOT&E  had  completed. 

•  The  Navy  began  the  IDECM  Block  IV  integrated  testing  in 
July  2013  and  completed  the  planned  testing  in  April  2014. 

-  The  integrated  tests  included  laboratory  testing  at  the 
Navy’s  Electronic  Combat  System  Evaluation  Laboratory, 
Point  Mugu,  California,  against  two  classified  threats  as 
well  as  a  dense  emitter  scenario,  and  flight  testing  at  ECR 
and  NTTR.  DOT&E  will  use  operationally  relevant  data 
from  the  integrated  testing  period  in  conjunction  with  data 
from  the  FOT&E  to  assess  the  IDECM  Block  IV  system. 
The  results  will  be  reported  in  DOT&E’s  classified 
IDECM  Block  IV  FOT&E  report. 

•  In  May  2014,  the  Navy  accomplished  a  limited-scope 
maintenance  demonstration  at  Naval  Air  Station 
China  Lake,  California.  The  results  will  be  included  in 
DOT&E’s  classified  IDECM  Block  IV  FOT&E  report. 

•  On  June  16,  2014,  the  Navy  accomplished  an  OTRR  for 
IDECM  Block  IV  FOT&E.  DOT&E  concurred  with  the 
Navy  in  commencing  operational  testing,  with  the  following 
caveats: 

-  DOT&E  expressed  concern  about  the  ambitious  schedule 
for  completing  flight  test  at  NTTR  in  only  two  range 
periods.  DOT&E  recommended  the  Navy  plan  for  a  third 
range  period. 

-  DOT&E  expressed  concern  that  the  Navy  had  only 
two  F/A-18C/D  aircraft  available  for  test,  which  could 
create  schedule  delays  if  F/A-18C/D  maintenance 
problems  occurred.  DOT&E  recommended  at  least 
three  aircraft  be  available  to  support  testing  since  many  of 
the  runs  required  at  least  two  aircraft  per  test  event. 

-  DOT&E  expressed  concern  that  there  was  the  potential 
the  schedule  would  need  to  be  extended  since  threat 
hardware  at  both  ECR  and  NTTR  had  been  experiencing 
failures. 

•  The  FOT&E  began  in  June  2014  and  is  scheduled  to  end 
in  February  2015.  The  Navy  is  conducting  testing  in 
accordance  with  the  DOT&E-approved  test  plan.  Numerous 


schedule  delays  have  occurred  due  to  threat  hardware 
problems  on  the  test  ranges,  aircraft  maintenance  problems, 
and  limited  fleet  aircraft  availability  to  support  testing. 

•  The  Navy  has  convened  an  F/A-18  wingman  compatibility 
working  group  that  has  members  from  multiple  Navy 
program  offices  to  investigate  and  resolve  deficiencies 
associated  with  the  aircraft  radar,  which  may  be  caused  by 
other  sub-systems  such  as  IDECM. 

Assessment 

•  DOT&E  assessed  the  IDECM  Block  IV  system  was  ready 
for  operational  test  at  the  OTRR.  The  Navy  had  adequately 
addressed  most  of  the  software  immaturity  and  interoperability 
deficiencies  at  that  time. 

•  Laboratory  testing  was  adequate,  but  using  higher  fidelity  radar 
cross  section  (RCS)  data  for  the  F/A-18  would  provide  more 
operationally  realistic  results. 

•  The  Navy  corrected  the  deficiency  caused  by  interaction 
between  the  ALR-67(V)2  and  (V)3  radar-waming  receivers  and 
IDECM  Block  IV  system,  which  caused  false  threat  symbols 

to  be  displayed.  However,  the  Navy  deferred  correcting  the 
deficiency  in  which  the  APG-79  radar  is  falsely  identified  to 
the  ALQ-214(V)4  by  the  ALR-67(V)2  and  (V)3  radar-warning 
receivers  to  a  wingman  compatibility  working  group  composed 
of  multiple  program  offices. 

•  DOT&E  will  produce  a  classified  FOT&E  report  assessing 
system  operational  effectiveness  and  suitability  after  the 
conclusion  of  the  IDECM  Block  IV  FOT&E. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  adequately 
addressed  some  previous  recommendations;  however,  the 
following  remain  outstanding: 

IDECM  System 

1 .  The  Navy  should  restructure  and  reorganize  the  complex  and 
poorly  organized  IDECM  system  software  code.  This  will 
minimize  potential  software  problems  yet  to  be  discovered 
and  simplify  future  modifications. 

2.  The  Navy  should  develop  hardware  and/or  software  changes 
to  provide  pilots  with  correct  indications  of  whether  a  decoy 
was  completely  severed.  This  recommendation  does  not 
apply  to  the  F/A-18C/D  installation  since  that  installation 
does  not  include  a  towed  decoy. 

3.  The  Navy  should  investigate  the  effects  of  IDECM  on  threat 
missile  fuses. 

Electronic  Warfare  Warfighting  Improvements 

4.  In  coordination  with  the  Defense  Intelligence  Agency, 
the  Navy  should  update  the  threat  lethal  radii  and/or  the 
evaluation  processes  that  are  used  to  determine  whether 
simulated  shots  are  hits  or  misses. 

•  FY  14  Recommendation. 

1.  The  Navy  should  use  the  highest  fidelity  F/A-18  RCS  data 
available  when  accomplishing  analysis  of  laboratory  testing, 
and  develop  accredited  RCS  models  that  account  for  the 
entire  F/A-18  airframe  configuration. 
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Joint  High  Speed  Vessel  (JHSV) 


Executive  Summary 

•  The  Navy  completed  IOT&E  and  LFT&E  of  the  Joint  High 
Speed  Vessel  (JHSV)  in  January  2014.  Additionally,  the 
Navy  conducted  the  first  of  two  FOT&E  events  in  June  and 
October  2014.  The  first  event  consisted  of  mooring  operations 
with  the  Mobile  Landing  Platform  with  the  Core  Capability 
Set  (MLP  (CCS)).  The  second  test  event  will  consist  of 
launching  and  retrieving  the  seal  delivery  vehicle  from  the 
JHSV  while  at  sea. 

•  DOT&E  issued  the  classified  combined  IOT&E  and  LFT&E 
report  on  July  17,  2014,  and  found  JHSV  effective  and 
suitable  with  the  following  limitations: 

-  JHSV  did  not  meet  its  fully  loaded  transit  speed  and 
distance  requirement  (600  short  tons  at  35  knots 
for  1,200  nautical  miles).  Testing  results  support 
600  short  tons  at  31  knots  for  858  nautical  miles. 

-  USNS  Spearhead  (JHSV-1)  had  an  unexplained  weight 
growth  of  over  47  short  tons  (specific  ship  weights  may 
vary  hull  to  hull),  which  contributed  to  the  fully  loaded 
transit  range  deficit  since  it  limited  the  amount  of  fuel  the 
ship  coidd  carry  without  exceeding  maximum  draft. 

-  JHSV  is  slightly  limited  in  self-deployment  range,  with  test 
results  supporting  a  4,000  nautical  mile  unrefueled  transit 
compared  to  the  4,700  nautical  mile  requirement. 

-  Small  boat  operations  from  JHSV  are  limited  to  Sea  State  2 
or  less  until  procedures  are  identified  to  allow  higher  Sea 
State  boat  operations. 

-  JHSV’s  cybersecurity  is  not  satisfactory  because  of 
numerous  significant  Information  Assurance  deficiencies. 

-  Initial  observations  and  assessment  indicate  deficiencies 
exist  with  skin-to-skin  mooring  operations  with  MLP 
(CCS).  In  the  first  test,  several  mooring  lines  parted, 
precluding  completion  of  the  test  event.  In  the  second  test, 
the  mooring  line  issue  was  resolved  but  the  JHSV  ramp 
suffered  a  casualty,  precluding  completion  of  the  test. 

-  JHSV  is  survivable  only  if  used  in  a  non-hostile 
environment.  The  ship  has  no  self-defense  capability  other 
than  crew-served  weapons;  it  requires  augmentation  with 

a  Navy  self-protection  team  (Embarked  Security  Team), 
and  is  built  to  commercial,  high-speed  craft  specifications 
that  do  not  include  hull,  equipment,  or  personnel  protection 
features  necessary  to  survive  weapons  effects.  The  ship 
has  only  rudimentary  capability  for  crew  protection  in 
chemical  or  biological  threat  environments. 

-  Although  JHSV  testing  shows  overall  ship  operation 
suitability,  the  ship  service  diesel  electric  generators  did 
not  meet  its  anticipated  reliability. 

System 

•  The  JHSV  is  a  high-speed,  shallow-draft  surface  vessel 
designed  for  intra-theater  transport  of  personnel  and  medium 


payloads  of  cargo  for  the  joint  force.  It  bridges  the  gap 
between  large  capacity  low-speed  sealift  and  small  capacity 
high-speed  airlift. 

•  JHSV  is  a  redesign  of  a  commercial  catamaran  capable 
of  accessing  relatively  austere  ports.  Classified  as  a  non 
combatant,  JHSV  has  limited  self-protection  capability. 
Design  characteristics  include  the  following: 

-  Propulsion  provided  by  four  water  jets  powered  by  diesel 
engines 

-  At-sea  refueling  capability 

-  Support  for  3 12  embarked  troops  for  up  to  96  hours  or 
104  troops  for  14  days 

-  Integrated  ramp  capable  of  load/off-load  of  military 
vehicles  including  combat-loaded  main  battle  tanks 
(M1A2) 

-  Flight  deck  with  helicopter  refueling  capability 

Mission 

Combatant  Commanders  will  use  the  JHSV  to  support  the 
flexible,  agile  maneuver  and  sustainment  of  combat  forces 
between  forward  operating  bases,  ports,  austere  littoral  access 
points,  and  the  sea  base.  Specifically,  Combatant  Commanders 
may  employ  the  JHSV  in  a  transport/resupply  role  in  benign, 
non-hostile  environments  to: 

•  Rapidly  transport  medium  payloads  of  Marine  Corps  or  Army 
cargo  and  combat-ready  troops  over  intra-theatre  distances 
between  shore  nodes 

•  Deliver  troops,  combat-loaded  vehicles,  and  equipment  ready 
to  be  employed,  requiring  only  ports  with  pier  or  quay  wall 
access  and  no  other  infrastructure 

•  Support  sustainment  of  forces  between  forward  operating 
bases,  ports,  and  austere  littoral  access  points  that  would  be 
prohibitive  for  larger  ships  to  access 

Major  Contractor 

Austal  USA  -  Mobile,  Alabama 
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Activity 

•  The  Navy’s  Commander,  Operational  Test  and  Evaluation 
Force  and  Marine  Corps  Operational  Test  and  Evaluation 
Activity  conducted  testing  on  USNS  Spearhead  (JHSV-1), 
USNS  Choctaw  Comity  (JHSV-2),  and  USNS  Millinocket 
(JHSV-3)  during  the  following  test  periods: 

-  Post  Delivery  Test  and  Trials  (PDT&T),  February  2013 
through  July  2013 

-  Dedicated  IOT&E  and  live  fire  testing,  July  2013  through 
August  2013 

-  PDT &T,  August  2013  through  J anuary  2014 

-  Partial  FOT&E,  June  and  October  2014 

•  These  tests  were  conducted  at  sea  while  transiting  from  the 
manufacturing  facility  in  Mobile,  Alabama,  to  their  home  port 
Joint  Expeditionary  Base  Little  Creek  in  Fort  Story,  Virginia; 
in  the  Virginia  Capes  area;  and  while  in  port  at  the  Norfolk 
Naval  Base,  Virginia.  The  FOT&E  events  were  conducted  in 
Long  Beach  harbor,  California,  in  June  2014,  and  at  sea  off 
Camp  Pendleton,  California,  in  October  2014. 

•  Personnel  from  the  Combatant  Craft  Division  of  Naval 
Surface  Warfare  Center  Carderock  Division  assisted  in  testing 
of  the  Rigid  Hull  Inflatable  Boat  (RH1B)  launch  and  recovery 
with  Sea  State  3  in  September  2013. 

•  Personnel  from  the  Naval  Surface  Warfare  Center  (NSWC), 
Port  Hueneme,  California,  conducted  an  underway 
replenishment  (fuel  receive  only)  ship  qualification  on 
USNS  Choctaw  County  both  in  port  at  Norfolk  Naval  Base 
and  underway  in  October  2013. 

•  Naval  aviators  from  Norfolk  Virginia  Naval  Air  Station 
operated  an  MH-60S  helicopter  off  the  JHSV  to  complete  the 
final  operational  test.  Both  the  JHSV  helicopter  fueling  and 
power  systems,  which  Naval  Air  Systems  Command  testers 
had  initially  found  to  be  deficient  during  tests  in  2013,  were 
retested  after  completion  of  required  repairs  and  modifications. 

•  On  May  19,  2014,  DOT&E  approved  the  JHSV  FOT&E  test 
plan.  FOT&E  consists  of  two  test  events:  demonstration 
of  the  transfer  capability  of  Marine  Corps  vehicles  between 
JHSV  and  MLP  (CCS)  while  at  sea  during  day  and  night 
(interface  tests),  and  the  launch  and  recovery  of  the  seal 
delivery  vehicle  from  the  JHSV  while  at  sea. 

-  The  Navy  attempted  to  conduct  a  day  and  night  interface 
test,  mooring  of  the  JHSV  to  the  MLP  (CCS)  vessel, 
during  the  week  of  June  23,  2014.  USNS  Millinocket 
(JHSV-3)  moored  skin-to-skin  with  USNS  Montford  Point 
(MLP-1)  and  Marine  Corps  vehicles  transited  back  and 
forth  during  daylight.  The  vehicle  transfer  in  daylight 
was  completed  successfully,  but  five  mooring  lines  broke, 
prompting  cancellation  of  the  night  test. 

-  The  Navy  attempted  a  second  interface  test  on 
October  29,  2014,  at  sea  off  Camp  Pendleton,  California. 
The  mooring  line  issue  was  resolved,  but  the  JHSV  ramp 
suffered  a  casualty,  prompting  cancellation  of  further 
vehicles  transfer  after  recovering  the  first  vehicle  and 
precluding  completion  of  the  test. 

•  All  tests  were  conducted  in  accordance  with  a 
DOT&E-approved  test  plan. 


Assessment 

•  The  classified  DOT&E  combined  IOT&E  and  LFT&E  report 
states  that  the  JHSV  is  effective  and  suitable  with  some 
limitations.  The  assessment  is  inclusive  of  activities  from 
FY13  and  FY14.  Further  details  of  the  following  assessment 
are  identified  in  the  classified  report. 

•  JHSV  is  capable  of  fueling  at  sea. 

•  The  JHSV  ramp  can  accommodate  all  anticipated  vehicles  to 
a  pier/quay  wall  and  to  a  Roll  On/Roll  Off  Discharge  Facility 
(floating  causeway)  during  Sea  State  1  conditions,  as  normally 
found  in  protected  harbors  (the  ramp  was  not  designed  for 
higher  sea  states). 

•  JHSV  cannot  achieve  the  required  23  knot,  4,700  nautical  mile 
light-ship  self-deployment  transit.  Calculations  show  it  will 
be  short  by  682  nautical  miles  if  it  started  with  a  90  percent 
starting  fuel  load  and  was  allowed  to  use  all  but  the  final 
10,000  gallons  in  storage. 

•  JHSV  cannot  make  the  required  35  knot,  1,200  nautical  mile 
fully  loaded  (600  short  tons)  transit.  Calculations  indicate 

it  can  make  a  31  knot,  858  nautical  mile  transit.  Longer  and 
faster  transits  will  require  refueling  from  a  tanker. 

•  There  is  an  unexplained  47.7  short  ton  growth  in  the  JHSV-1 
ship  weight  that  represents  13,700  gallons  of  fuel  that  cannot 
be  loaded  without  exceeding  maximum  draft  (maximum  total 
ship  and  cargo  weight). 

•  With  an  embarked  security  team,  which  includes  both 
personnel  and  weapons,  JHSV  can  engage  a  moving  surface 
threat.  However,  testers  identified  the  following  safety 
problem:  0.50  caliber  gun  operators  are  vulnerable  to  falling 
overboard  during  high  seas  or  radical  ship  maneuvers  due  to 
the  lowering  of  lifelines  during  gun  use. 

•  JHSV  manning  and  facilities  can  accommodate  handling  of 
all  required  unarmed  helicopters  including  providing  fuel  and 
power. 

•  The  JHSV  crew  demonstrated  day  and  night  vertical 
replenishment  with  MH-60,  MH-53,  and  MV-22. 

•  After  completion  of  required  repairs  and  modifications,  the 
JHSV  crew  successfully  retested  the  JHSV  helicopter  fueling 
and  power  systems,  which  Naval  Air  Systems  Command 
testers  had  initially  found  to  be  deficient  during  tests  in  2013. 

•  The  JHSV  crew  demonstrated  efficient  loading,  securing,  and 
unloading  of  Marines  and  their  equipment  from  II  Marine 
Expeditionary  Force,  including  their  rolling  stock. 

•  The  JHSV  crew  demonstrated  the  96-hour  requirement  for 
transporting  and  feeding  312  combat  troops  by  supporting 
354  people  over  that  period. 

•  In  August  2013,  the  crew  only  successfully  demonstrated  the 
capability  to  launch  two  11-meter  RHIBs  within  40  minutes 
in  Sea  State  2  (wave  heights  up  to  2  feet).  The  requirement  is 
to  conduct  this  operation  in  Sea  States  up  to  3  (wave  heights 
up  to  4  feet).  Looking  for  Sea  State  3,  the  Navy  conducted 

a  second  attempt  in  September  2013.  However,  the  Ship’s 
Master  cancelled  attempts  to  launch  a  RHIB  in  Sea  State  3  due 
to  safety  concerns. 
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•  The  ship’s  crew  demonstrated  the  underway  requirement  to 
move  a  27,000-pound  container  from  the  mission  bay  to  the 
flight  deck  and  back  in  Sea  State  3. 

•  The  JHSV’s  organic  Cargo  Loading  Trailer  is  not  effective 
for  loading  20-foot  long  metal  storage  containers.  During  the 
IOT&E,  the  test  team  took  five  hours  to  connect  the  Cargo 
Loading  Trailer  with  a  storage  container  and  failed  to  load  it 
aboard  the  ship.  A  20-foot  storage  container  was  loaded  at 
the  ship  home  port  where  ramp  access  from  the  pier  was  less 
restrictive. 

•  The  ship  service  diesel  electric  generators  installed  in  JHSV 
demonstrated  poor  reliability  during  the  test  period.  Their 
target  Mean  Time  Between  Failure  was  2,496  hours,  but  was 
measured  to  be  only  208  hours. 

•  Cybersecurity  testing  found  numerous  significant  Information 
Assurance  vulnerabilities  and  an  absence  of  written 
instructions  and  procedures  to  guide  shipboard  network 
administrators. 

•  Initial  observations  and  assessment  indicate  deficiencies 
exist  with  skin-to-skin  mooring  operations  with  MLP  (CCS). 
Although  mooring  line  issues  that  arose  in  early  testing  were 
resolved,  the  JHSV  ramp  does  not  seem  to  be  able  to  handle 
the  small  but  continual  relative  movement  of  the  two  ships 
when  moored  skin-to-skin. 

•  JHSV  is  survivable  only  if  used  in  a  non-hostile  environment. 
The  ship  has  no  self-defense  capabilities  other  than 

crew-  served  weapons;  it  requires  augmentation  with  a  Navy 
self-protection  team;  and  is  built  to  commercial,  high-speed 
craft  specifications  that  do  not  include  hull  and  equipment 
hardening  or  personnel  protection  features  necessary  to  survive 
weapons  effects. 

•  Vulnerability  modeling  identified  a  significant  risk  of 
casualties  if  JHSV  is  used  in  Major  Combat  Operations  and 
put  in  harm’s  way.  Small  civilian  crews  operate  the  ships 
and  are  trained  and  equipped  to  recover  from  small-scale 
casualties,  not  those  likely  in  major  combat  operations. 


Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  addressed 
two  of  the  six  FY13  recommendations.  The  Navy  still  needs 
to  determine  the  best  self-deployed  transit  speed  to  achieve 
the  4,700  nautical  mile  unrefueled  range  requirement,  a 
transit  speed  that  allows  for  a  600  short  ton  load  delivery  to 
1,200  nautical  miles,  and  an  outfitted  weight  for  each  hull  to 
enable  mission  planners  to  characterize  fully  loaded  transit 
capability.  The  Navy  also  still  needs  to  evaluate  design 
improvements  identified  during  the  Total  Ship  Survivability 
Trials  and  implement  those  that  will  enhance  the  ship’s 
survivability. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Review  and  modify  tactics,  techniques,  and  procedures  to 
safely  launch  RHIBs  in  Sea  States  greater  than  Sea  State  2. 

2.  Consider  a  replacement  for  the  Cargo  Loading  Trailer  if 
a  JHSV  is  utilized  routinely  to  transport  20-foot  storage 
containers 

3.  Implement  a  reliability  growth  program  for  the  ship  service 
diesel  electric  generators. 

4.  Resolve  and  retest  the  significant  Information  Assurance 
vulnerabilities  identified  in  the  classified  DOT&E  combined 
IOT &E  and  LFT &E  report. 

5.  Address  lack  of  written  instructions  and  procedures  to  guide 
shipboard  network  administrators. 

6.  Provide  safety  lanyards  and  harnesses  for  embarked  security 
team  members  that  man  gun  mounts.  Additionally,  provide 
hands-free  communication  devices  to  help  coordinate  firing 
engagements. 

7.  Investigate  casualty  problem  with  JHSV’s  ramp  that 
occurred  during  interface  test  with  MLP  (CCS)  in 
October  2014.  If  necessary,  reevaluate  the  need  for  at  sea 
skin-to-skin  operations  between  JHSV  and  MLP  (CCS). 

8.  Complete  JHSV  FOT&E  with  the  launch  and  recovery  of 
the  seal  delivery  vehicle. 
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Joint  Standoff  Weapon  (JSOW) 


Executive  Summary 

•  The  Navy  accomplished  additional  integrated  testing  of 
the  AGM-154C-1  Joint  Standoff  Weapon  (JSOW)  variant 
during  FY14.  Problems  identified  during  FY12-13  integrated 
testing  resulted  in  follow-on  integrated  testing  in  late  FY14 
and  pushed  operational  testing  to  FY15.  The  JSOW  C-l 
operational  testing  is  scheduled  to  begin  in  mid-FY15. 

•  Preliminary  results  to  date  indicate: 

-  Weapon  impact  accuracy  for  moving  maritime  targets  is 
well  within  the  accuracy  requirement  value,  and  accuracy 
performance  against  stationary  land  targets  has  been 
maintained. 

-  The  JSOW  C-l  Mean  Flight  Hours  Between  Operational 
Mission  Failure  (MFHBOMF)  remains  below  the 
requirement  value,  primarily  the  result  of  software-driven 
problems,  but  is  showing  progress  towards  meeting  the 
requirement  value. 

-  The  pilot-vehicle  interface  (PVI)  has  improved,  but  there 
remain  some  minor  challenges  the  aircrew  must  work 
around  for  successful  mission  execution. 

•  Testing  of  the  implemented  updates  to  the  JSOW  software  to 
address  these  problems  validate  the  use  of  developmental  and 
integrated  test  data  for  DOT&E’s  operational  evaluation  of 
JSOW  C-l. 

System 

•  The  AGM-154  JSOW  family  uses  a  common  and  modular 
weapon  body  capable  of  carrying  various  payloads.  The 
JSOW  is  a  1,000-pound  class,  air-to-surface  glide  bomb 
intended  to  provide  low  observable,  standoff  precision 
engagement  with  launch  and  leave  capability.  All  variants 
employ  a  tightly  coupled  GPS/Inertial  Navigation  System. 

•  AGM- 1 54 A  (JSOW  A)  payload  consists  of  1 45  BLU-97/B 
combined  effects  submunitions. 

•  AGM-154C  (JSOW  C)  utilizes  an  imaging  infrared  seeker 
and  its  payload  consists  of  an  augmenting  charge  and 
follow-through  bomb  that  can  be  set  to  detonate  both 
warheads  simultaneously  or  sequentially. 

•  AGM- 154 A  and  AGM-154C  are  fielded  weapons  and  no 
longer  under  DOT&E  oversight.  AGM-154C-1  (JSOW  C-l) 


adds  moving  maritime  target  capability  and  the  two-way  strike 
common  weapon  datalink  to  the  baseline  AGM-154C  weapon. 

Mission 

•  Combatant  Commanders  use  JSOW  A  to  conduct  pre-planned 
attacks  on  soft  point  and  area  targets  such  as  air  defense  sites, 
parked  aircraft,  airfield  and  port  facilities,  command  and 
control  antennas,  stationary  light  vehicles,  trucks,  artillery,  and 
refinery  components. 

•  Combatant  Commanders  use  JSOW  C  to  conduct  pre-planned 
attacks  on  point  targets  vulnerable  to  blast  and  fragmentation 
effects  and  point  targets  vulnerable  to  penetration  such  as 
industrial  facilities,  logistical  systems,  and  hardened  facilities. 

•  Units  will  use  JSOW  C-l  to  conduct  attacks  against  moving 
maritime  targets  and  have  the  ability  to  retarget  weapons  post 
launch.  JSOW  C-l  will  retain  the  JSOW  C  legacy  capability 
against  stationary  land  targets. 

Major  Contractor 

Raytheon  Company,  Missile  Systems  -  Tucson,  Arizona 


Activity 

•  The  Navy  conducted  developmental  and  integrated  testing 
in  accordance  with  a  DOT&E-approved  Test  and  Evaluation 
Master  Plan  for  the  JSOW  C-l. 

•  The  Navy  was  scheduled  to  complete  operational  testing 
following  the  previous  FY12-13  integrated  testing,  but  it  was 
delayed  to  2QFY15  due  to  problems  with  the  complicated 


PVI  and  integration  with  the  F/A-18E/F  H8  Operational  Flight 
Program  (OFP). 

•  The  Navy  implemented  79  software  change  requests,  of  which 
34  addressed  major  PVI  and  integration  problems.  This 
activity  required  a  change  of  approximately  4  percent  of  the 
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123,000  software  lines  of  codes  in  the  guidance  electronics 
unit. 

•  The  Navy  conducted  nine  additional  integrated  captive-carry 
tests  between  1QFY14  and  4QFY14  to  verify  the  software 
fixes  and  integration  with  the  F/A-18E/F  H10  OFP.  The 
Navy  has  one  free-flight  test  against  a  moving  maritime  target 
planned  for  2QFY15. 

•  Results  from  the  developmental  and  integrated  testing  will 
support  an  Operational  Test  Readiness  Review  (OTRR)  in 
2QFY15.  The  Navy  has  rescheduled  JSOW  C-l  operational 
testing  for  FY15  following  the  OTRR. 

Assessment 

•  The  Navy  accomplished  additional  integrated  testing  of  JSOW 
C-l  in  FY14.  Preliminary  results  to  date  indicate: 

-  Weapon  impact  accuracy  for  moving  maritime  targets 
continues  to  be  well  within  the  accuracy  requirement  value 
and  accuracy  performance  against  stationary  land  targets 
has  been  maintained. 

-  JSOW  C-l  MFHBOMF  is  below  the  requirement  value. 
This  is  primarily  the  result  of  software-driven  problems. 
With  the  migration  from  F/A-18  H8  OFP  to  H10,  the 
MFHBOMF  is  showing  progress  towards  meeting  the 
requirement  value.  Achieving  adequate  assessment  of 
MFHBOMF  during  operational  testing  is  an  area  of 
moderate  risk. 


-  The  Navy  has  reduced  the  complexity  of  the  PVI  in  the 
F/A-18E/F  H10  OFP.  There  remain  minor  PVI  challenges 
that  could  prevent  successful  mission  execution.  These 
challenges  can  be  effectively  overcome  with  proper 
training  prior  to  employment.  This  is  an  area  of  low  risk 
during  operational  testing. 

•  Testing  of  the  implemented  updates  to  the  JSOW  software  to 
address  these  problems  validate  the  use  of  developmental  and 
integrated  test  data  for  DOT&E’s  operational  evaluation  of 
JSOW  C-l. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  partially 
addressed  the  previous  recommendations.  The  Navy  has 
demonstrated  a  reduction  in  software-driven  failures  during 
the  extended  integrated  testing  phase.  While  they  have 
significantly  reduced  the  complex  PVI,  their  plan  will  not 
fully  address  this  issue  until  the  F/A-18E/F  H12  OFP  release, 
scheduled  for  FY17. 

•  FY 14  Recommendation. 

1.  The  Navy  should  continue  to  reduce  the  PVI  complexity 
between  the  JSOW  C-l  and  the  F/A-18  Super  Hornet  to 
permit  successful  mission  execution. 
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LHA-6  New  Amphibious  Assault  Ship 


Executive  Summary 

•  The  Navy  accepted  delivery  of  LHA-6  in  April  2014, 
seven  months  behind  schedule. 

•  The  Navy  identified  a  budget  shortfall  that  will  likely  prohibit 
the  completion  of  IOT&E  by  the  acquisition  program 
threshold  of  October  2016  for  reaching  Initial  Operational 
Capability.  In  addition,  the  late  delivery  of  the  ship  reduces  its 
availability  for  operational  testing  prior  to  its  deployment  in 
FY17. 

•  The  Navy  and  Marine  Corps  Operational  Test  Agencies 
conducted  the  first  phase  of  an  Early  Operational  Assessment 
for  LHA-8  in  May  2014. 

System 

LHA-6  is  a  large-deck  amphibious  ship  designed  to  support 
a  notional  mix  of  fixed-  and  rotary-wing  aircraft  consisting 
of  12  MV-22s,  6  F-35B  Joint  Strike  Fighters  (Short  Take 
Off/ Vertical  Landing  variant),  4  CH-53Es,  7  AH-ls/UH-ls,  and 
2  embarked  H-60  Search  and  Rescue  aircraft,  or  a  load  out  of 
20  F-35Bs  and  2  embarked  H-60  Search  and  Rescue  aircraft. 

Key  ship  features  and  systems  include: 

•  Greater  aviation  storage  capacity  and  an  increase  in  the 
size  of  the  hangar  bay,  which  is  necessary  to  accommodate 
the  increased  maintenance  requirements  of  the  F-35B  and 
the  MV-22.  Additionally,  two  maintenance  areas  with 
high-overhead  clearance  are  incorporated  into  the  design  of 
the  ship  to  accommodate  wings-open  MV-22  maintenance. 

•  Shipboard  medical  spaces  reduced  by  approximately  two 
thirds  compared  to  contemporary  LHDs  to  expand  the  hangar 
bay. 

•  A  Ship  Self-Defense  System  Mk  2-based  combat  system  with 
the  following  seven  major  components. 

-  The  Ship-Self  Defense  System  Mk  2  Mod  4B  control  and 
decision  system  supports  the  integration  and  control  of 
most  other  combat  system  elements 

-  The  ship’s  AN/SPS-48E  and  AN/SPS-49A  air  search  radars 
and  the  AN/SPQ-9B  horizon  search  radar 

-  USG-2  Cooperative  Engagement  Capability  radar  tracking 
systems 

-  The  Rolling  Airframe  Missile  and  the  Evolved  SeaSparrow 
Missile,  with  the  NATO  SeaSparrow  Mk  9  Track 
Illuminators 

-  The  AN/SLQ-32B(V)2  electronic  warfare  system  with  the 
Nulka  electronic  decoy-equipped  Mk  53  Decoy  Launching 
System 

-  The  Phalanx  Close-in  Weapon  System  for  air  and  small 
boat  defense 

-  The  Mk  38  Mod  2  Gun  Weapon  System  for  small  boat 
defense 


•  Propulsion  is  provided  by  two  marine  gas  turbine  engines, 
two  electric  auxiliary  propulsion  motors,  and  two  controllable 
pitch  propellers.  Six  diesel  generators  provide  electric  power. 

•  Command,  Control,  Communications,  Computers,  and 
Intelligence  facilities  and  equipment  to  support  Marine  Corps 
Landing  Force  operations  are  part  of  the  program  of  record. 

•  LHA-6  and  LHA-7  do  not  have  well  decks,  which  are 
traditionally  used  to  move  large  volumes  of  heavy  equipment 
needed  for  amphibious  operations.  LHA-8  and  later  ships  will 
have  a  well  deck. 

Mission 

The  Joint  Maritime  Component  Commander  will  employ  LHA-6 

to: 

•  Be  the  primary  aviation  platform  within  an  Amphibious  Ready 
Group  with  space  and  accommodations  for  Marine  Corps 
vehicles,  cargo,  ammunition,  and  more  than  1,600  troops 

•  Serve  as  an  afloat  headquarters  for  a  Marine  Expeditionary 
Unit,  Amphibious  Squadron,  or  other  Joint  Force  commands 
using  its  Command,  Control,  Communications,  Computers, 
and  Intelligence  facilities  and  equipment 

•  Accommodate  elements  of  a  Marine  Expeditionary  Brigade 
when  part  of  a  larger  amphibious  task  force 

•  Carry  and  discharge  combat  service  support  elements  and 
cargo  to  sustain  the  landing  force 

Major  Contractor 

Huntington  Ingalls  Industries,  Ingalls  Shipbuilding 

Division  -  Pascagoula,  Mississippi 
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Activity 

•  The  Navy  accepted  delivery  of  LHA-6  in  April  2014.  This 
was  seven  months  late  and  has  the  potential  to  affect  the 
planned  IOT&E  schedule. 

•  In  September  2014,  the  Navy  identified  a  budget  shortfall  that 
will  prevent  the  completion  of  IOT&E  prior  to  the  acquisition 
program  threshold  of  October  2016.  The  Navy  is  investigating 
alternate  courses  of  action  to  mitigate  this  problem. 

•  The  Navy  and  Marine  Corps  Operational  Test  Agencies 
conducted  the  first  phase  of  an  Early  Operational  Assessment 
(EOA)  design  review  for  LHA-8  in  May  2014. 

-  The  Navy  desired  to  conduct  this  test  to  identify  design 
risks  prior  to  releasing  the  Request  for  Proposal. 

-  The  test  was  not  conducted  in  accordance  with  the 
DOT&E-approved  test  plan  because  many  of  the  specified 
subject  matter  experts  were  not  available  and  the  review 
was  not  conducted  in  a  facility  that  permitted  classified 
work.  Hence,  the  ship’s  self-defense  capability  was  not 
reviewed. 

•  A  second  phase  of  the  EOA  that  was  originally  planned  to 
examine  the  ship’s  amphibious  attack  and  air  assault  mission 
capability  in  late  1QFY15  is  currently  under  review  due  to  the 
withdrawal  of  funding. 

Assessment 

•  Because  of  a  seven-month  delay  in  the  ship’s  delivery  and 
resultant  delays  in  the  scheduling  of  the  ship’s  self-defense 
weapon  systems  test,  the  LHA-6  program  is  unlikely  to 
complete  IOT&E  by  the  acquisition  program  threshold  of 
October  2016. 

•  If  the  $4.5  Million  allocated  in  the  Test  and  Evaluation  Master 
Plan  for  the  amphibious  warfare  IOT&E  event  in  FY16  is 
not  restored  to  permit  the  event  to  occur  as  planned,  the 
operational  testing  that  can  be  conducted  prior  to  acquisition 
program  threshold  for  reaching  Initial  Operating  Capability 
will  be  inadequate  to  determine  the  ship’s  effectiveness  and 
suitability.  The  Navy  is  investigating  how  to  conduct  the 
amphibious  warfare  portion  of  IOT&E  prior  to  the  ship’s 
deployment  in  2017. 

•  The  Total  Ship  Survivability  Trial,  which  contributes  to  the 
survivability  assessment  of  the  ship,  was  planned  to  occur 
during  the  amphibious  warfare  event  to  minimize  the  cost  of 
the  test  program.  If  this  now  has  to  be  scheduled  to  occur  by 
itself,  it  may  require  a  larger  test  budget. 

•  The  first  phase  of  the  LHA-8  EOA  was  too  limited  to  identify 
shortfalls,  if  any,  in  the  ship’s  design. 


•  The  LHA-6  Ship  Self-Defense  System  (SSDS)  has 
demonstrated  capability  against  some  classes  of  anti-ship 
cruise  missile  (ASCM)  threats.  However,  based  on  combat 
systems  testing  on  other  platforms,  it  is  unlikely  that  LHA-6 ’s 
SSDS  Mk  2-based  combat  system  will  meet  the  ship’s 
Probability  of  Raid  Annihilation  requirement  against  all 
classes  of  ASCMs. 

•  The  Fire  Control  Loop  Improvement  Program  (FCLIP)  was 
initiated  to  correct  some  combat  system  deficiencies  related  to 
self-defense  against  ASCMs  discovered  using  the  Self-Defense 
Test  Ship  and  has  the  potential  to  mitigate  some  of  the  likely 
shortfalls  in  the  ship’s  self-defense  capability.  However, 
FCLIP  has  been  only  partially  implemented. 

•  Phase  One  of  the  FCLIP  has  been  completed,  but  not 
operationally  tested.  Phases  Two  and  Three  of  the  proposed 
program  have  not  been  funded. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  either 
addressed  or  established  a  process  through  which  to  address 
most  of  the  previous  recommendations.  The  Navy  has  not 
fully  resolved  the  following  four  recommendations  to: 

1.  Correct  systems  engineering  deficiencies  related  to  SSDS 
Mk  2-based  combat  systems  and  other  combat  system 
deficiencies  so  that  LHA-6  can  satisfy  its  Probability  of 
Raid  Annihilation  requirement. 

2.  Evaluate  the  survivability  improvement  recommendations 
resulting  from  the  analysis  of  the  LHA-6  design  for 
incorporation  into  the  LHA-7  design. 

3.  Implement  improvements  to  the  SSDS  Mk  2-based  combat 
system  and  test  those  changes  during  FOT&E. 

4.  Make  the  Multi-Stage  Supersonic  Target  available  to 
support  an  assessment  of  the  LHA-6  Probability  of  Raid 
Annihilation  requirement. 

•  FY 14  Recommendations.  The  Navy  should: 

1.  Allocate  sufficient  resources  to  permit  the  IOT&E  and 
Total  Ship  Survivability  Trial  to  be  conducted  as  currently 
planned,  or  seek  approval  of  a  modified  course  of  action 
that  allows  all  operational  testing  objectives  to  be  satisfied. 

2.  Review  and  revise  the  ship’s  post-delivery  schedule  to 
assure  that  sufficient  time  is  available  for  the  completion  of 
IOT&E  prior  to  its  first  deployment. 
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Littoral  Combat  Ship  (LCS) 


Executive  Summary 

•  During  FY14,  the  Navy  conducted  both  developmental  testing 
and  operational  testing  of  the  Freedom  class  Littoral  Combat 
Ship  (LCS)  seaframe  and  Increment  2  Surface  Warfare  (SUW) 
Mission  Package  aboard  USS  Fort  Worth  (LCS  3).  The  2014 
operational  testing  identified  shortcomings  in  air  defense, 
reliability,  and  endurance,  and  significant  vulnerabilities  in 
cybersecurity.  When  equipped  with  the  Increment  2  SUW 
Mission  Package,  LCS  3  was  able  to  defeat  a  small  number 
of  Fast  Inshore  Attack  Craft  under  the  particular  conditions 
specified  by  the  Navy’s  reduced  incremental  requirement 

and  after  extensive  crew  training  and  tailoring  of  the  tactics 
described  in  Navy  doctrine;  however,  testing  conducted  to 
date  has  not  been  sufficient  to  demonstrate  LCS  capabilities  in 
more  stressing  scenarios  consistent  with  existing  threats. 

•  The  core  combat  capabilities  of  the  Independence  class  variant 
seaframe  remain  largely  untested.  Developmental  testing  has 
focused  on  evaluating  the  performance  of  the  seaframe  and  the 
Increment  1  Mine  Countermeasures  (MCM)  Mission  Package, 
with  multiple  deficiencies  identified. 

-  The  MCM  Mission  Package  has  not  yet  demonstrated 
sufficient  performance  to  achieve  the  Navy’s  minimal 
Increment  1  requirements.  Although  the  ship’s  and  its 
crew’s  ability  to  launch  and  recover  Remote  Multi-Mission 
Vehicles  (RMMVs)  has  improved,  LCS  has  had  difficulty 
establishing  and  maintaining  reliable  communications  with 
the  RMMV,  and  the  RMMV  continues  to  exhibit  reliability 
problems.  The  current  communications  systems  also  do 
not  support  bottom  mine  identification  beyond  the  horizon. 

-  Attempts  to  demonstrate  the  sequence  of  events  necessary 
for  an  LCS  to  complete  end-to-end  mine  clearance 
operations  have  been  limited  by  low  operator  proficiency, 
software  immaturity,  system  integration  problems,  and 
poor  Remote  Minehunting  System  (RMS)/RMMV 
reliability. 

-  During  a  shore-based  assessment,  the  Airborne  Mine 
Neutralization  System  (AMNS)  did  not  meet  the  Navy’s 
requirement  for  mine  neutralization  success.  Failures 
of  the  host  MF1-60  aircraft’s  systems  and  its  associated 
Airborne  MCM  kit  severely  limited  AMNS  availability. 
Frequent  loss  of  fiber-optic  communications  between 
the  aircraft  and  the  neutralizer  was  the  primary  cause  of 
unsuccessful  attack  runs.  Both  problems  increase  the 
time  needed  to  conduct  LCS-based  AMNS  operations  and 
reduce  the  ship’s  sustained  area  coverage  rate. 

•  LCS  is  not  expected  to  be  survivable  in  high-intensity  combat 
because  its  design  requirements  accept  the  risk  that  the  ship 
must  be  abandoned  under  circumstances  that  would  not 
require  such  an  action  on  other  surface  combatants.  Although 
the  ship  incorporates  capabilities  to  reduce  susceptibility  to 


Freedom  Variant  (LCS  1 ) 


Independence  Variant  (LCS  2) 


attack,  previous  testing  of  analogous  capabilities  demonstrates 
it  cannot  be  assumed  LCS  will  not  be  hit  in  high-intensity 
combat. 

-  While  both  seaframe  variants  are  fast  and  highly 
maneuverable,  they  are  lightly  armed  and  possess  no 
significant  offensive  capability  without  the  planned 
SUW  Increment  4  Mission  Package  or  Increment  2 
Anti-Submarine  Warfare  (ASW)  Mission  Package. 

•  Equipment  reliability  problems  have  degraded  the  operational 
availability  of  both  LCS  classes.  The  Navy  reports  that 
recent  reliability  improvements  made  to  the  affected  seaframe 
components  have  led  to  improved  operational  availability; 
however,  that  improvement  has  not  been  verified  in 
operational  testing. 

•  At  the  request  of  the  Secretary  of  Defense,  DOT&E  prepared 
in  October  2014  an  independent  assessment  of  the  combat 
capabilities  and  survivability  of  the  alternative  concepts 

for  a  new  small  surface  combatant  (SSC)  developed  by  the 
Navy’s  SSC  Task  Force.  Using  the  Task  Force’s  results, 
that  assessment  found  that  only  major  modifications  to  the 
existing  LCS  design,  or  a  new  ship  design,  could  provide  the 
multi-mission  combat  capabilities  and  survivability  features 
found  in  a  modern  frigate. 
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System 

Seaframes 

•  The  LCS  is  designed  to  operate  in  the  shallow  waters  of 
the  littorals  that  can  constrain  the  ability  of  larger  ships  to 
maneuver. 

•  The  Navy  planned  to  acquire  a  total  of  52  LCSs;  however, 
in  a  February  24,  2014,  memorandum,  the  Secretary  of 
Defense  announced  that  no  new  contract  negotiations 
beyond  32  ships  will  go  forward  and  directed  the  Navy  to 
submit  alternative  proposals  to  procure  a  capable  and  lethal 
SSC,  generally  consistent  with  the  capabilities  of  a  frigate. 
In  December,  he  approved  the  Navy’s  recommendation  to 
modify  the  existing  LCS  designs  for  the  remaining  20  ships. 

•  The  Navy  is  currently  procuring  two  variants  of  LCS 
seaframes: 

-  The  Freedom  class  (LCS  1,  3,  5,  and  follow-on 
odd-numbered  ships)  is  a  semi-planing  monohull  design 
constructed  of  steel  (hull)  and  aluminum  (deckhouse) 
with  two  steerable  and  two  fixed-boost  water  jets  driven 
by  a  combined  diesel  and  gas  turbine  main  propulsion 
system. 

-  The  Independence  class  (LCS  2,  4,  6,  and  follow-on 
even-numbered  ships)  is  an  aluminum  trimaran  design 
with  two  steerable  water  jets  driven  by  diesel  engines 
and  two  steerable  water  jets  driven  by  gas  turbine 
engines. 

•  Common  design  specifications  include: 

-  Sprint  speed  in  excess  of  40  knots,  draft  of  less 
than  20  feet,  and  an  unrefueled  range  in  excess  of 
3,500  nautical  miles  at  14  knots 

-  Accommodations  for  up  to  98  personnel 

-  A  Common  Mission  Package  Computing  Environment 
for  mission  package  control 

-  Hangars  sized  to  embark  MH-60R/S  and  Vertical 
Take-Off  Unmanned  Aerial  Vehicles  (VTUAVs) 

-  Mk  1 10  57  mm  gun 

•  The  designs  have  different  core  combat  systems  to 
provide  command  and  control,  situational  awareness,  and 
self-defense  against  anti-ship  cruise  missiles  and  surface 
craft. 

-  Freedom  Class  Variant: 

■  COMBATSS-21,  an  Aegis-based  integrated  combat 
weapons  system  with  a  TRS-3D  (SPS-75)  air/surface 
search  radar 

■  Ship  Self-Defense  System  Rolling  Airframe  Missile 
(RAM)  system  (one  21-cell  launcher) 

■  Soft  Kill  Weapon  System 

■  DORNA  gunfire  control  system  with  an 
electro-optical/infrared  sensor  to  control  the 
Mk  110  57  mm  gun. 

-  Independence  Class  Variant: 

■  Integrated  Combat  Management  System  (derived 
from  Dutch  TACT1COS  system)  with  a  Sea  Giraffe 
(SPS-77)  air/surface  search  radar 

■  One  Mk  15  Mod  31  SeaRAM  launcher  (integrates 
the  search,  track,  and  engagement  scheduler  of  the 


Close-in  Weapon  System  with  an  11-round  RAM 
launcher  assembly) 

■  ALEX  (Automatic  Launch  of  Expendables)  System 
(off-board  decoy  countermeasures) 

■  Sea  Star  SAF1RE  electro-optical/infrared  systems  for 
57  mm  gun  fire  control. 

Mission  Packages 

•  LCS  is  designed  to  host  a  variety  of  individual  warfare 
systems  (mission  modules)  assembled  and  integrated  into 
interchangeable  mission  packages.  The  Navy  currently 
plans  to  field  MCM,  SUW,  and  ASW  Mission  Packages. 

A  mission  package  provides  the  seaframes  with  capability 
for  a  single  or  “focused”  mission.  Multiple  individual 
programs  of  record  involving  sensor  and  weapon  systems 
and  off-board  vehicles  make  up  the  individual  mission 
modules. 

SUW  Mission  Package 

-  Increment  1  includes: 

■  Gun  Mission  Module  (two  Mk  46  30  mm  guns) 

■  Aviation  Module  (embarked  MH-60R) 

-  Increment  2  adds: 

■  Maritime  Security  Module  (small  boats) 

-  Increment  3  adds: 

■  Surface-to-Surface  Missile  Module  Increment  I, 
employing  the  AGM  114L  Longbow  HELLFIRE 
missile 

■  One  MQ-8B  Fire  Scout  VTUAV  to  the  Aviation 
Module 

-  Increment  4  adds: 

■  Surface-to-Surface  Missile  Module  Increment  II  to 
replace  Increment  I 

MCM  Mission  Package 

-  Increment  1  includes: 

■  Remote  Minehunting  Module,  consisting  of  two 
RMMVs  and  three  AN/AQS-20A/B  sensors.  The 
Navy  is  considering  plans  to  incorporate  an  improved 
sensor  (AN/AQS-20B)  in  the  current  increment; 
otherwise,  the  AN/AQS-20B  will  most  likely  be 
delivered  in  a  future  increment. 

■  Near  Surface  Detection  Module  consisting  of  two 
Airborne  Laser  Mine  Detection  Systems  (ALMDS). 
The  Navy  is  also  developing  pre-planned  product 
improvements  that  it  expects  to  incorporate  in  a  future 
increment. 

■  Airborne  Mine  Neutralization  Module  consisting  of 
two  AMNS.  In  Increment  1,  the  AMNS  does  not 
include  a  near-surface  mine  neutralization  capability; 
the  Navy  plans  to  develop  this  capability  in  a  future 
increment. 

■  Aviation  Module  consisting  of  an  MH-60S  Block  2B 
or  subsequent  Airborne  Mine  Countermeasures 
(AMCM)  Helicopter  outfitted  with  an  AMCM  system 
operator  workstation  and  a  tether  system. 
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-  Increment  2  adds: 

■  Coastal  Battlefield  Reconnaissance  and  Analysis 
(COBRA)  Block  I  system  and  one  MQ-8B  VTUAV 
for  daytime  unmanned  aerial  tactical  reconnaissance 
to  detect  and  localize  mine  lines  and  obstacles  in  the 
beach  zone. 

-  Increment  3  adds: 

■  Unmanned  Influence  Sweep  System  to 
activate  acoustic-,  magnetic-,  and  combined 
acoustic/magnetic-initiated  volume  and  bottom  mines 
in  shallow  water  so  they  self-destruct. 

-  Increment  4  adds: 

■  COBRA  Block  II  system,  which  retains  Block  I 
capability  and  adds  night-time  minefield  and  obstacle 
detection  capability  and  day/night  detection  capability 
in  the  surf  zone. 

■  Knifefish  Unmanned  Undersea  Vehicle,  an  untethered, 
autonomous  underwater  vehicle,  employing  a 
low-frequency  broadband  sonar  sensor  to  detect, 
classify,  and  identify  volume  and  bottom  mines  in 
shallow  water. 

ASW Mission  Package  (only  Increment  2) 

-  Torpedo  Defense  and  Countermeasures  Module 
(Lightweight  Tow  torpedo  countermeasure) 

-  ASW  Escort  Module  (Multi-Function  Towed  Array  and 
Variable  Depth  Sonar) 

-  Aviation  Module  (embarked  MH-60R  and  MQ-8B  Fire 
Scout  VTUAV)  (Inclusion  of  Fire  Scout  has  reportedly 
been  deferred  because  of  fiscal  constraints.) 

Mission 

•  The  Maritime  Component  Commander  will  employ  LCS  to 
conduct  MCM,  ASW,  or  SUW  tasks  depending  on  the  mission 


package  fitted  into  the  seaframe.  Commanders  can  employ 
LCS  in  a  maritime  presence  role  in  any  configuration  because 
of  capabilities  inherent  to  the  seaframe.  With  the  Maritime 
Security  Module,  installed  as  part  of  the  SUW  Mission 
Package,  the  ship  can  conduct  Visit,  Board,  Search,  and 
Seizure  maritime  interception  operations. 

•  The  Navy  can  employ  LCS  alone  or  in  company  with  other 
ships.  The  Navy’s  Concept  of  Operations  for  LCS  anticipates 
that  the  ship’s  primary  operational  role  will  involve  preparing 
the  operational  environment  for  joint  force  assured  access 

to  critical  littoral  regions  by  conducting  MCM,  ASW,  and 
SUW  operations,  possibly  under  an  air  defense  umbrella  as 
determined  necessary  by  the  operational  commander. 

Major  Contractors 

•  Freedom  Class  Variant  (LCS  1,  3,  5,  and  follow-on 
odd-numbered  ships) 

-  Prime:  Lockheed  Martin  Maritime  Systems  and 
Sensors  -  Washington,  District  of  Columbia 

-  Shipbuilder:  Marinette  Marine  -  Marinette,  Wisconsin 

•  Independence  Class  Variant  (LCS  2,  4,  6,  8,  and  follow-on 
even-numbered  ships) 

-  Prime  for  LCS  2  and  LCS  4:  General  Dynamics 
Corporation  Marine  Systems,  Bath  Iron 
Works  -  Bath,  Maine 

-  Prime  for  LCS  6,  LCS  8,  and  follow-on  even  numbered 
ships:  Austal  USA- Mobile,  Alabama 

-  Shipbuilder:  Austal  USA- Mobile,  Alabama 

•  Mission  Packages 

-  Future  Mission  Package  Integration  contract  awarded  to 
Northrop  Grumman  -  Los  Angeles,  California 


Activity 

LCS  Program 

•  DOT&E  published  an  Early  Fielding  Report  providing  an 
assessment  of  the  LCS  seaframes  and  mission  packages  in 
December  2013. 

•  DOT&E  forwarded  a  response  to  the  reporting  requirement 
in  the  FY14  Conference  Report  (113-66)  accompanying 
H.R.  3304,  the  National  Defense  Authorization  Act  for 
FY14,  to  the  Navy  for  inclusion  with  the  Navy’s  response 
on  April  4,  2014. 

•  DOT&E  will  publish  an  assessment  of  the  results  of 
operational  testing  of  the  Freedom  class  seaframe  and  SUW 
Mission  Package  (Increments  1  and  2)  in  2QFY15. 

Seaframes 

•  Freedom  Class  Variant: 

-  USS  Freedom  (LCS  1)  returned  to 

San  Diego,  California,  from  operations  in  the 
Western  Pacific  in  December  2013. 


-  After  receiving  modifications  needed  to  host  the  ASW 
Mission  Package,  USS  Freedom  (LCS  1)  participated  in 
engineering  tests  designed  to  test  the  integration  of  the 
ASW  mission  modules  into  the  Freedom  class  seaframe. 

-  The  Navy  conducted  developmental  testing  of  the 
Freedom  class  variant  seaframe  aboard  USS  Fort  Worth 
(LCS  3)  in  1Q-2QFY14. 

-  The  Navy  conducted  operational  testing  of  the  Freedom 
class  variant  seaframe  aboard  USS  Fort  Worth  (LCS  3) 
in  March  and  April  2014.  The  testing  was  conducted  in 
accordance  with  a  DOT&E-approved  test  plan. 

•  Independence  Class  Variant: 

-  USS  Independence  (LCS  2)  hosted  a  scheduled  phase 
of  developmental  testing  focused  on  the  integrated 
seaframe  and  Increment  1  MCM  Mission  Package 
operations  in  October  2014. 
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-  USS  Independence  (LCS  2)  participated  in  a  phase  of 
developmental  testing  designed  to  evaluate  RMMV 
launch  and  recovery  operations  using  modified 
hardware  and  revised  procedures  and  communications 
between  the  LCS  and  RMMVs  using  the  Multi- Vehicle 
Communications  System. 

-  The  Navy  conducted  Heavy  Weather  developmental 
testing  in  early  FY14.  Subsequently,  the  Navy 
discovered  cracks  in  the  welds  at  the  base  of  support 
stanchions  located  in  the  mission  bay  and  imposed  a 
weight  limit  on  the  launch  and  recovery  system  for 
USS  Independence  (LCS  2)  and  USS  Coronado  (LCS  4). 

-  The  Navy  placed  USS  Coronado  (LCS  4)  in  commission 
in  April  2014. 

-  The  Board  of  Inspection  and  Survey  (INSURV) 
conducted  Final  Contract  Trials  aboard  USS  Coronado 
(LCS  4)  in  June  2014. 

-  USS  Coronado  (LCS  4)  completed  a  series  of  basic 
seaframe  developmental  tests,  including  57  mm  gunnery 
events,  before  starting  her  post-shake-down  availability 
in  1QFY15. 

SUW  Mission  Package 

•  The  Navy  conducted  developmental  testing  of  the 
Increment  2  SUW  Mission  Package  aboard  USS  Fort  Worth 
(LCS  3)  in  1Q-2QFY14. 

•  The  Navy  conducted  an  initial  phase  of  operational 
testing  of  the  Increment  2  SUW  Mission  Package  aboard 
USS  Fort  Worth  (LCS  3)  in  March  and  April  2014.  This 
test  examined  the  Freedom  class  variant’s  self-defense 
capability  against  small  swarms  of  high-speed  boats  and  its 
effectiveness  in  conducting  simulated  maritime  interdiction 
operations  that  required  the  crew  to  intercept  and  board  a 
vessel  suspected  of  transporting  contraband. 

•  The  testing  was  conducted  in  accordance  with  a 
DOT&E-approved  test  plan. 

MCM  Mission  Package 

•  In  1QFY 14,  the  Navy  completed  two  phases  of 
developmental  testing  (DT-IIG)  of  the  RMS  consisting 
of  a  version  4.2  (v4.2)  RMMV  and  an  AN/AQS-20A 
sensor  from  a  shore  base  at  the  contractor’s  facility  in 
West  Palm  Beach,  Florida.  A  third  phase  of  testing 
described  by  the  RMS  Test  and  Evaluation  Master  Plan 
as  an  opportunity  to  assess  risk  of  the  interfaces  with  the 
LCS,  including  cybersecurity,  was  not  conducted.  The 
Navy  cited  lack  of  LCS  availability  as  the  rationale  for 
cancellation  of  this  phase,  but  incompatibility  of  the  v4.2 
RMMV  with  LCS  was  also  a  factor.  In  3QFY14,  the  Navy 
conducted  dockside  and  at-sea  developmental  testing  to 
verify  correction  of  the  RMMV  launch,  handling,  and 
recovery  system  and  communications  deficiencies  observed 
in  FY13  developmental  testing.  Finally,  in  1QFY15,  the 
Navy  conducted  the  last  scheduled  phase  of  the  Increment  1 
MCM  Mission  Package  developmental  test  DT-B2  aboard 
USS  Independence  (LCS  2).  This  phase  was  the  first 


time  that  RMS  and  the  airborne  MCM  Mission  Package 
components  had  operated  together  off  an  LCS. 

•  The  Navy  conducted  Phase  A  of  an  operational  assessment 
of  the  AMNS  in  3QFY14  with  the  MH-60S  helicopter 
operating  from  Naval  Air  Station  Oceana,  Virginia.  The  test 
included  the  use  of  explosive  neutralizers  and  inert  training 
neutralizers  against  inert  moored  and  bottom  mine  targets. 

•  The  Navy  conducted  Phase  B  of  the  AMNS  operational 
assessment  to  focus  on  shipboard  integration  and  the 
system’s  suitability,  but  was  also  able  to  collect  some 
limited  effectiveness  data.  This  phase  of  test  was  conducted 
aboard  USS  Independence  (LCS  2)  in  1QFY15  during 
Increment  1  MCM  Mission  Package  developmental  testing. 
Because  some  deficiencies  in  mission  package  performance 
discovered  earlier  had  not  been  corrected,  some  deviations 
from  the  approved  test  plan  occurred;  those  deficiencies  are 
discussed  in  the  assessment  section  below. 

•  The  Navy  completed  Phase  B  (LCS-based  phase)  of  the 
test  in  1QFY15  aboard  USS  Independence  (LCS  2)  during 
Increment  1  MCM  Mission  Package  developmental  testing. 
The  test  examined  system  effectiveness  and  the  shipboard 
suitability  of  the  MH-60S  helicopter  equipped  with 
ALMDS. 

•  The  Navy  postponed  a  scheduled  operational  assessment 
of  COBRA  Block  1  after  an  Antares  rocket  exploded 
just  after  lift-off  from  the  Wallops  Island  launch  pad  on 
October  28,  2014.  Although  all  test  preparations  had  been 
completed,  both  MQ-8B  Fire  Scout  VTUAVs  that  were  to 
host  the  COBRA  system  during  the  test  suffered  shrapnel 
damage  from  the  rocket  explosion.  The  Navy  has  not  yet 
established  a  new  date  for  the  operational  assessment. 

ASW  Mission  Package 

•  The  Navy  conducted  Advanced  Development  Model 
(ADM)  testing  of  the  ASW  Escort  Module  aboard  LCS  1  in 
September  2014.  Testing  was  focused  on  the  integration  of 
the  Variable  Depth  Sonar  and  Multi-Function  Towed  Array 
with  the  LCS  platform,  to  include  pull  stresses  and  stern 
door  effectiveness  with  penetrating  systems.  Testing  also 
included  some  long-range  passive  and  active  ASW  search 
in  deep  water  against  a  U.S.  nuclear  submarine.  Testing 
was  highly  scripted,  which  is  appropriate  for  early  system 
integration  efforts,  but  the  results  cannot  be  used  to  make 
any  assessment  of  operational  performance  under  realistic 
combat  conditions.  The  test  was  conducted  with  full 
knowledge  of  the  target  submarine’s  position  throughout 
the  test,  and  the  operators  focused  their  search  only  in  the 
region  where  the  submarine  was  known  to  be. 

LFT&E 

•  The  Navy  conducted  a  Total  Ship  Survivability  Trial 
(TSST)  on  USS  Fort  Worth  (LCS  3)  in  accordance  with  the 
DOT&E-approved  trial  plan. 

•  DOT&E  provided  an  interim  survivability  assessment  of 
both  LCS  designs  in  the  Early  Fielding  Report  issued  on 
December  9,  2013. 
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•  As  part  of  the  response  to  the  reporting  requirement  in 
the  FY14  Conference  Report  (113-66)  accompanying 
H.R.  3304,  the  National  Defense  Authorization  Act 
for  FY 14,  DOT&E  provided  the  Navy  a  review  of  the 
survivability  testing,  modeling,  and  simulation  conducted 
to  date  on  the  two  seaframes  and  an  assessment  of  the 
expected  survivability  of  LCS  in  the  context  of  the 
planned  employment  of  LCS  as  described  in  the  concept  of 
operations. 

SSC  Study 

•  At  the  request  of  the  Secretary  of  Defense,  DOT&E 
prepared  in  October  2014  an  independent  assessment  of 
the  combat  capabilities  and  survivability  of  the  alternative 
concepts  for  a  new  SSC  developed  by  the  Navy’s  SSC  Task 
Force.  The  DOT&E  assessment  is  a  classified  document. 

Assessment 

This  assessment  is  based  on  information  from  DOT&E’s 
observations  of  post-delivery  testing  and  trial  events,  fleet 
operations,  developmental  test  data,  results  provided  by  the 
Navy  program  offices,  operational  assessments  of  some  mission 
systems,  and  operational  testing  of  the  Freedom  class  seaframe 
with  the  Increment  1  and  2  SUW  Mission  Packages. 

LCS  Program 

•  The  Navy  intends  to  field  LCS  capabilities  incrementally 
as  mission  package  systems  mature  and  become  ready 
for  fleet  use.  Additionally,  the  Navy  directed  changes 

to  the  seaframe  designs  based  on  the  results  of  early 
developmental  testing  and  operations. 

-  The  Navy  has  indicated  that  the  seaframe  designs  will 
be  stabilized  in  the  third  ship  of  each  variant  (LCS  5  and 
LCS  6). 

-  Since  the  Navy  expects  each  increment  to  deliver 
significant  increases  in  mission  capability,  the  approved 
Test  and  Evaluation  Master  Plan  anticipates  an 
appropriately-designed  phase  of  OT&E  on  all  delivered 
mission  package  increments  on  each  seaframe  variant. 
The  details  of  the  testing  to  be  accomplished  for  later 
Increments  of  mission  package  capability  will  be  decided 
when  the  content  of  those  later  increments  are  defined  by 
the  Navy. 

-  An  initial  phase  of  operational  testing  was  completed  in 
FY14  for  the  Freedom  class  variant  seaframe  and  SUW 
Mission  Package  only,  but  the  final  phases  will  not  be 
completed  until  the  FY19  timeframe. 

Seaframes 

•  While  both  seaframe  variants  are  fast  and  highly 
maneuverable,  they  are  lightly  armed  and  possess  no 
significant  offensive  capability  without  the  planned 
Increment  4  SUW  Mission  Package  or  the  Increment  2 
ASW  Mission  Package. 

-  In  comparison  to  other  Navy  ships,  the  LCS  seaframes 
have  relatively  modest  air  defense  capabilities; 


however,  their  air  defense  capabilities  cannot  be 
characterized  fully  until  tests  on  LCS  5  and  LCS  6  (the 
production-representative  seaframes)  and  the  Navy’s 
unmanned  Self-Defense  Test  Ship  provide  data  for  the 
Navy  Probability  of  Raid  Annihilation  high-fidelity 
modeling  and  simulation  analyses  in  FY18.  The  Navy 
plans  to  test  the  Independence  class  variant’s  capability 
to  defeat  unmanned  aerial  vehicles  and  slow-flying 
aircraft  in  FY15. 

-  The  Freedom  class  seaframe’s  surface  self-defense 
capability  was  operationally  tested  in  FY14  (see  below) 
and  the  Independence  class  seaframe’s  capability  is 
scheduled  to  be  tested  in  FY15  aboard  USS  Coronado 
(LCS  4). 

-  The  seaframes  include  no  systems  designed  to  detect 
torpedo  attacks  or  mines  without  the  appropriately 
configured  mission  packages  installed. 

•  Crew  size  can  limit  the  mission  capabilities,  combat 
endurance,  and  recoverability  of  the  ships.  The  Navy 
continues  to  review  manning  to  determine  appropriate 
levels,  and  is  adding  20  berths  to  all  seaframes.  The 
increased  berthing  supports  small  increases  in  the  size 
of  the  core  crew,  mission  package  detachments,  and  the 
aviation  detachment. 

•  Freedom  Class  Variant  (LCS  1  and  3): 

-  Although  not  all  aspects  of  operational  effectiveness 
and  operational  suitability  could  be  examined  during 
the  2014  operational  testing,  that  testing  identified 
shortcomings  in  air  defense,  reliability,  and  endurance, 
and  significant  vulnerabilities  in  cybersecurity. 

-  Cybersecurity  testing  conducted  during  operational 
testing  aboard  LCS  3  uncovered  significant 
vulnerabilities  in  the  ship’s  capability  to  protect  the 
security  of  information  and  prevent  malicious  intrusion. 
Limited  cybersecurity  testing  conducted  during  a 
2012  Quick  Reaction  Assessment  aboard  LCS  1  also 
found  vulnerabilities. 

-  Tracking  events  conducted  during  operational  testing 
aboard  LCS  3  demonstrated  that  in  some  scenarios  the 
SPS-75  (TRS-3D)  air  search  radar  is  unable  to  detect 
and  track  some  types  of  air  threats  in  operationally 
realistic  environments.  Tracking  performance 
improved  significantly  when  the  LCS  received  tracking 
information  via  datalink  from  a  nearby  Aegis  destroyer. 
The  lack  of  an  integrated  electronic  support  measures 
system  limits  the  ship’s  capability  to  make  best  use  of  its 
inventory  of  RAM  surface-to-air  missiles. 

-  Critical  equipment  required  to  support  ship  operations, 
core  mission  functions,  and  mission  package  operations 
is  unreliable.  The  ship’s  crew  does  not  have  adequate 
training  and  technical  documentation  to  troubleshoot 
equipment  failures;  the  Navy  lacks  repair  parts  for  some 
critical  systems;  and  the  Navy’s  plan  for  distribution 

of  the  maintenance  workload  among  the  ship’s  crew, 
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shore-based  Navy  support  organizations,  and  technical 
experts  from  other  organizations  is  immature.  The 
operational  availability  of  shipboard  systems  in 
10  of  12  categories  examined  met  or  exceeded  Navy 
requirements,  however,  failures  of  critical  propulsion  and 
maneuvering  and  Total  Ship  Computing  Environment 
systems  forced  the  ship  to  return  to  port  for  repairs  that, 
respectively,  caused  42  and  36  days  of  downtime  during 
the  period  of  data  collection  during  operational  testing. 
Excluding  scheduled  maintenance  periods,  LCS  3  was 
fully  mission  capable  less  than  25  percent  of  the  time 
during  that  period. 

-  During  operational  testing,  LCS  3  did  not  demonstrate 
that  it  could  achieve  the  Navy  requirement  for  fuel 
endurance  (operating  range)  at  the  prescribed  transit 
speed  or  at  sprint  speed.  Information  provided  by 

the  Navy  indicated  that  between  91  and  92  percent 
of  the  ship’s  total  diesel  fuel  (F-76)  tank  capacity 
would  actually  be  available  for  use  since  some  room 
must  be  left  for  expansion  when  the  tanks  are  filled,  a 
portion  of  the  tanks’  volume  is  filled  with  piping  and 
structural  members,  and  a  small  amount  of  fuel  remains 
inaccessible  when  the  tanks  are  emptied.  Based  on 
fuel  consumption  data  collected  during  the  test,  the 
ship’s  operating  range  at  14.4  knots  is  estimated  to  be 
approximately  1,961  nautical  miles  (Navy  requirement: 
3,500  nautical  miles  at  14  knots)  and  the  operating 
range  at  43.6  knots  is  approximately  855  nautical 
miles  (Navy  requirement:  1,000  nautical  miles  at 
40  knots).  In  an  emergency,  the  ship  could  use  its 
aviation  fuel  (F-44)  to  extend  the  transit  and  sprint 
ranges  by  360  and  157  nautical  miles,  respectively.  The 
shortfall  in  endurance  may  limit  the  flexibility  of  the 
ship’s  operations  in  the  Pacific  and  place  a  heavier  than 
anticipated  demand  on  fleet  logistics. 

-  Operational  testing  confirmed  earlier  observations  that, 
except  for  the  ships’  lack  of  endurance,  the  Freedom 
class  variant  is  well-suited  for  Maritime  Security 
Operations.  LCS  3  readily  demonstrated  the  capability 
to  position,  launch,  and  recover  the  11 -meter  boats 
included  in  the  SUW  Mission  Package  when  the  launch, 
recovery,  and  handling  system  is  operational. 

-  The  ship’s  Mk  1 10  57  mm  gun  system  performed 
reliably  during  operational  testing,  and  the  ship  was 
able  to  demonstrate  the  core  capability  for  self-defense 
against  a  small  boat  in  two  valid  trials.  The  Navy 
attempted  to  collect  additional  data  from  swarm 
presentations,  but  the  data  were  invalid.  The  57  mm 
gun  failed  to  achieve  a  mission  kill  during  one  swarm 
presentation,  and  the  target  killed  by  the  57  mm  gun 
during  a  second  swarm  presentation  had  previously  been 
engaged  by  30  mm  guns. 

-  The  Freedom  class  LCS  has  sufficient  aviation  facilities 
and  meets  Navy  requirements  to  safely  launch,  recover. 


and  handle  all  appropriate  aircraft  while  operating  in 
Sea  State  4  conditions.  However,  the  ship  frequently 
experienced  difficulty  with  establishing  and  maintaining 
a  Tactical  Common  Data  Link  with  the  aircraft  during 
the  FY14  operational  test.  The  crew’s  efforts  were 
hampered  by  an  antenna  failure  and  the  total  lack 
of  technical  documentation  on  the  operation  and 
maintenance  of  the  datalink. 

-  The  LCS  3  anchoring  system  could  not  securely  anchor 
the  ship  in  an  area  with  a  bottom  composed  of  sand  and 
shells.  Despite  repeated  efforts,  the  ship  was  unable  to 
set  the  anchor.  It  appears  that  the  anchor  and  chain  are 
too  light  and  there  are  too  many  friction  points  along 
the  anchor  chain’s  internal  path  from  the  chain  locker  to 
the  hawse  pipe  to  allow  the  anchor  and  chain  to  pay  out 
smoothly. 

-  The  fenders  designed  to  guide  the  11 -meter  Rigid 
Hull  Inflatable  Boats  included  in  the  SUW  Mission 
Package  during  launch  and  recovery  are  fragile  and 
occasionally  sheared  off  when  impacted  by  the  boats 
during  operational  testing.  Although  the  fenders  have 
undergone  several  redesigns,  they  are  not  yet  strong 
enough  to  sustain  such  impacts. 

•  Independence  Class  Variant  (LCS  2): 

-  DOT&E  still  has  no  data  to  assess  the  core  mission 
capabilities  of  the  Independence  class  variant  seaframe. 

-  The  USS  Independence  (LCS  2)  crew  encountered 
multiple  problems  with  the  twin-boom  extensible  crane 
(TBEC)  and  other  mission  package  support  systems 
during  initial  developmental  testing  of  the  MCM  Mission 
Package.  Since  then,  the  vendor  has  improved  the 
TBEC,  and  the  Navy  has  made  changes  to  the  RMMV 
launch  and  recovery  hardware.  Developmental  testing  in 
August  2013,  May  2014,  and  October  2014  demonstrated 
that  the  ship’s  capability  to  launch  and  recover  the 
RMMV  has  improved  because  of  crew  training,  but  it  is 
not  yet  clear  that  launch  and  recovery  can  be  completed 
routinely  without  problems. 

-  In  the  past,  availability  of  the  USS  Independence 
(LCS  2)  to  support  testing  has  been  degraded  by 
equipment  failures,  including  problems  with  operator 
consoles,  power  generation  equipment,  components 
of  the  ship’s  computing  and  networking  equipment, 
propulsion  drive  train  components,  and  communications 
systems.  DOT&E  is  unable  to  evaluate  the  success 

of  Navy  efforts  to  improve  the  reliability  of  these 
systems.  In  September  and  October  2014,  the  start  of 
developmental  testing  of  the  MCM  Mission  Package  was 
delayed  by  LCS  air  conditioning  and  propulsion  system 
failures.  During  at-sea  testing,  observers  noted  that  LCS 
sometimes  experienced  difficulties  when  communicating 
with  a  simulated  Mine  Warfare  Commander  operating 
from  a  shore -based  command  center. 
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SUW  Mission  Package 

•  LCS  3  equipped  with  the  Increment  2  SUW  Mission 
Package  demonstrated  the  capability  to  defeat  a  small 
swarm  of  Fast  Inshore  Attack  Craft  under  the  conditions 
specified  in  the  Navy  requirement;  however,  the  crew 
received  extensive  hands-on  training  that  might  not  be 
available  to  crews  on  other  ships.  Testing  conducted  to  date 
has  not  been  sufficient  to  demonstrate  LCS  capabilities  in 
more  stressing  scenarios  consistent  with  existing  threats. 

•  The  SUW  Mission  Package  has  not  yet  been  tested  aboard 
an  Independence  class  LCS. 

•  The  30  mm  Gun  Mission  Modules  (GMM)  remain  prone 
to  jams  caused  by  separation  of  ammunition  links  and 
accumulation  of  spent  cartridges  in  the  ejection  path; 
however,  LCS  3  experienced  fewer  jams  during  operational 
testing  than  had  been  observed  in  past  developmental 
testing.  While  the  Navy  has  made  a  concerted  effort  to 
improve  ammunition  belts,  the  problem  was  not  entirely 
eliminated.  Ammunition  jams  interrupt  firing  but  can 
typically  be  cleared  in  a  few  minutes;  however,  they  are 
still  sufficiently  disruptive  to  cause  the  ship  to  maneuver  to 
bring  the  other  30  mm  GMM  to  bear  on  the  target. 

MCM  Mission  Package 

•  During  developmental  testing,  attempts  to  demonstrate 
the  sequence  of  events  necessary  for  an  LCS  to  complete 
end-to-end  mine  clearance  operations  have  been  limited 
by  low  operator  proficiency,  software  immaturity, 
system  integration  problems,  and  poor  reliability  of 
MCM  components  including  RMS/RMMV.  In  the  most 
recent  period  of  developmental  testing  in  1QFY15,  fleet 
operators  using  mission  package  tools  such  as  the  Organic 
Post  Mission  Analysis  (OPMA)  and  the  new  Contact 
Management  Tool  (CMT)  failed  to  convey  some  mine 
targets,  correctly  detected  by  the  RMS  in  an  initial  search 
pass,  to  the  AMNS  for  neutralization.  As  a  result,  fleet 
operators  were  unable  to  execute  operationally-realistic, 
end-to-end  mine  reconnaissance  and  clearance  without 
intervention  by  testers  with  knowledge  of  ground  truth 
target  positions.  The  Navy  continues  to  investigate  the 
root  cause  of  target  position  errors  and  incorrectly  dropped 
contacts;  unless  corrected,  these  problems  will  limit  LCS 
MCM  mission  effectiveness. 

•  During  developmental  testing,  the  operational  availability  of 
MCM  Mission  Package  systems  has  been  degraded  by  low 
reliability,  the  LCS  crew’s  limited  capacity  for  corrective 
maintenance,  and  the  ship’s  constrained  inventory  of 
repair  parts.  Testing  has  often  been  delayed  to  obtain  the 
assistance  of  shore -based  technicians  and  repair  parts  not 
available  onboard  LCS.  Left  uncorrected,  these  problems 
will  severely  limit  LCS’s  operational  capability  for  mine 
reconnaissance  and  clearance. 

•  Mission  package  minehunting  systems  (AN/AQS-20A 
and  ALMDS)  have  not  demonstrated  the  detection  and 
localization  capabilities  needed  for  an  LCS  equipped  with 
an  Increment  1  MCM  Mission  Package  to  meet  its  required 


sustained  area  coverage  rate.  During  developmental 
testing  and  a  shore -based  operational  assessment, 
AN/AQS-20A  contact  depth  (vertical  localization)  errors 
have  exceeded  Navy  limits  in  all  operating  modes. 

A  shore -based  operational  assessment  of  ALMDS 
showed  that  the  system  does  not  meet  Navy  detection 
requirements.  Both  systems  generate  a  large  number 
of  false  classifications  (objects  erroneously  classified 
as  mine-like).  Unless  eliminated  from  the  contact  list, 
these  false  classifications  require  identification  and 
neutralization  effort,  residt  in  the  expenditure  of  limited 
neutralizer  assets,  and  substantially  reduce  the  LCS 
sustained  area  coverage  rate.  As  an  alternative,  the  Navy 
has  implemented  tactics  that  require  multiple  search 
passes  over  the  same  area  to  minimize  the  number  of  false 
classifications  conveyed  for  identification/neutralization. 
Although  multiple  search  passes  also  reduce  the  LCS 
sustained  area  coverage  rate  relative  to  single  pass 
systems,  Navy  modeling  suggests  this  approach  is  less 
detrimental  to  MCM  timelines.  Whether  LCS  can  meet 
the  already-reduced  low  area  clearance  requirement  for 
the  Increment  1  Mission  Package  remains  in  question. 
Furthermore,  testing  has  not  yet  shown  whether  the 
goal  of  minimizing  AN/AQS-20A  false  classifications 
can  be  accomplished  without  also  eliminating  correct 
classifications  from  the  contact  list  and  degrading 
minehunting  performance. 

-  The  Navy  expected  to  correct  AN/AQS-20A 
deficiencies  prior  to  the  first  phase  of  operational 
testing  in  FY15  by  implementing  pre-planned  product 
improvements  (the  AN/AQS-20B  version  of  the 
sonar)  and  integrating  the  improved  sensor  into  the 
MCM  Mission  Package.  Delays  in  the  delivery  of 
AN/AQS-20B  prototypes  and  problems  discovered 

in  early  characterization  testing  in  FY14  leave  little 
time  to  complete  necessary  developmental  and 
operational  testing  of  the  AN/AQS-20B  prior  to  the 
planned  operational  test  of  LCS  equipped  with  the  first 
increment  of  the  MCM  Mission  Package  in  FY15. 

-  The  Navy  is  working  on  pre-planned  product 
improvements  to  improve  ALMDS  detection 
performance  and  reduce  the  frequency  of  receiver 
failures,  but  does  not  expect  to  integrate  these  changes 
into  the  first  increment  of  the  MCM  Mission  Package. 
Frequent  receiver  failures  continued  to  affect  ALMDS 
performance  during  an  experimental  deployment  to 
the  Navy’s  5th  fleet  and  recent  developmental  testing 
aboard  LCS  2.  During  LCS  developmental  testing,  the 
MH-60S  aircrew  was  also  unable  to  assess  ALMDS 
achieved  search/clearance  level  during  post-mission 
analysis.  Observations  from  5th  fleet  operators  also 
indicate  mission  planning  and  evaluation  tools  do  not 
adequately  support  ALMDS  mission  planning  and 
post-mission  clearance  estimates. 
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•  During  a  shore -based  operational  assessment  of  the 
AMNS  in  FY14,  AMNS  was  unable  to  achieve  the  Navy’s 
requirement  for  mine  neutralization  success  except  under 
limited  conditions  not  generally  expected  during  combat. 
Failures  of  the  host  MH-60S  aircraft’s  systems  and  its 
associated  AMCM  Mission  Kit  limited  AMNS  mission 
availability.  Frequent  loss  of  fiber-optic  communications 
between  the  aircraft  and  the  neutralizer  was  the  primary 
cause  of  unsuccessful  attack  runs.  Although  the  Navy 
attributed  the  failures  to  the  bottom  composition  (sand  and 
shells),  the  root  cause  of  these  failures  has  not  yet  been 
determined,  and  the  bottom  compositions  used  in  testing  are 
representative  of  realistic  operating  areas.  Both  problems 
negatively  affect  the  timeliness  of  LCS-based  AMNS 
operations  and  will  likely  reduce  the  ship’s  sustained  area 
coverage  rate. 

•  As  noted  earlier,  the  Independence  class  LCS  has  had 
difficulty  launching  and  recovering  the  RMMV  because 
of  the  vehicle’s  erratic  motion  in  the  ship’s  wake.  In  past 
developmental  testing,  violent  RMMV  yaw  and  roll  motions 
have  overstressed  and  damaged  the  launch  and  recovery 
hardware  and  resulted  in  damage  to  the  RMMV,  causing 
the  Navy  to  limit  handling  operations  to  when  sea  state 

is  less  than  3.  Following  changes  to  launch  and  recovery 
hardware,  procedures,  training,  and  RMMV  hardware, 
the  Navy  demonstrated  16  RMMV  launches  and  14 
RMMV  recoveries  during  23  days  at  sea  in  developmental 
testing  during  favorable  sea  state  conditions  in  1QFY15. 
Nonetheless,  the  most  recent  period  of  developmental 
testing  witnessed  several  instances  of  equipment  damage 
that  delayed  or  prevented  recovery  of  an  off-board  RMMV. 
Because  of  the  cracks  in  the  welds  at  the  base  of  support 
stanchions  located  in  the  mission  bay,  during  this  phase  of 
testing,  launch  and  recovery  operations  could  be  conducted 
only  when  wave-induced  loading  on  the  recovery  system 
(a  function  of  wave  height  and  period)  did  not  exceed 
32,000  pounds-force.  For  example,  a  wave  height  of  2  feet 
coupled  with  a  wave  period  of  2  seconds,  which  could 
occur  in  a  Sea  State  2,  would  have  precluded  RMMV 
recovery  until  calmer  sea  conditions  developed.  The  Navy 
revealed  they  are  making  design  changes  to  LCS  6  and  later 
seaframes  to  correct  the  problem  and  remove  the  weight 
limit.  LCS  2  and  LCS  4  will  be  corrected  during  the  next 
shipyard  availability.  This  problem  must  be  corrected  to 
ensure  safe  and  sustained  RMS  operations. 

-  No  RMMV  launch  and  recovery  operations  have  been 
conducted  aboard  a  Freedom  class  LCS  at  sea. 

-  Although  the  RMMV  can  search  autonomously  while 
operating  over  the  horizon  from  the  LCS,  it  can  currently 
only  conduct  operations  to  reacquire  and  identify 
bottom  mines  within  the  range  of  Ultra  High  Frequency 
communications.  This  limitation  will  complicate 
MCM  operations  in  long  shipping  channels,  and  may 
make  it  necessary  to  clear  a  series  of  LCS  operating 
areas  to  allow  MCM  operations  to  progress  along  the 


channel.  The  cleared  operating  areas  will  be  needed  to 
keep  the  LCS  and  its  crew  out  of  mined  waters.  The 
additional  effort  required  to  clear  these  LCS  operating 
areas  would  increase  the  demand  for  mine  clearance 
and  delay  attainment  of  strategic  objectives.  This 
issue  is  not  new  to  RMS;  however,  it  did  not  become 
operationally  significant  until  the  Navy  decertified  the 
MH-60S  helicopter  for  towing  MCM  devices,  including 
the  AN/AQS-20A/B  sensor.  The  RMS  communication 
range  limitation  was  not  an  operational  concern  when 
the  option  existed  for  the  helicopter  with  towed  sensor 
to  conduct  identification  operations  beyond  the  horizon. 
The  Navy  has  not  yet  identified  a  solution. 

-  RMS  reliability  problems  persisted  in  the  recent  phase 
of  developmental  testing  (1QFY15)  evidenced  in  part 
by  fewer  vehicle  recoveries  than  vehicle  launches. 
Problems  observed  include  the  inability  to  align 
the  system’s  inertial  navigational  unit,  intermittent 
communications,  a  lube  oil  pump  failure  that  caused  a 
mission  abort,  capture  latch  impairment  that  precluded 
shipboard  recovery  of  the  RMMV,  degraded  electro-optic 
identification  resulting  in  a  mission  abort  to  replace  the 
AN/AQS-20A  towed  body,  tow  cable  damage  following 
an  apparent  snag  that  rendered  the  system  inoperable  in 
the  assigned  mission  until  a  replacement  tow  cable  could 
be  installed  with  the  assistance  of  shore -based  support, 
and  multiple  incidents  of  AN/AQS-20A  stuck  fins  or  fin 
actuation  faults.  Although  the  Navy  demonstrated  more 
frequent  RMMV  launches  during  this  period  of  testing, 
continued  RMS  reliability  problems  limited  system 
minehunting  to  less  than  50  hours  during  the  3  weeks 
of  most  intensive  testing  (approximately  16  hours  per 
week). 

■  LCS  reliability  problems  also  forced  the  ship  to 
remain  in  port  for  repairs  instead  of  conducting  at-sea 
RMS  testing  as  planned.  Including  an  additional 
week  spent  in  port  for  LCS  repairs,  RMS  averaged 
approximately  12  hours  of  minehunting  per  week. 

This  result  is  consistent  with  the  assessment  of  RMS 
capability  DOT&E  provided  to  members  of  the 
Defense  Acquisition  Board  following  RMMV  v4.2 
and  AN/AQS-20A  testing  to  indicate  that  the  Navy 
had  not  yet  demonstrated  that  it  could  sustain 
operations  of  more  than  one  14-hour  RMMV  sortie 
per  week  (i.e.,  10  to  12  hours  of  RMS  minehunting 
per  week).  Unless  greater  minehunting  operating 
tempo  is  achieved,  the  Navy  will  not  meet  its  interim 
area  clearance  rate  requirements. 

■  The  Navy  reports  that  the  RMS  operated  for 
approximately  140  hours  during  LCS  developmental 
testing  in  1QFY15.  DOT&E ’s  preliminary  assessment 
of  test  data  identified  at  least  seven  RMS  failures 

that  precluded  vehicle  recovery,  required  sensor 
replacement,  or  required  assistance  from  shore -based 
support  contractors  to  restore  system  availability.  In 
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operational  testing,  these  failures  would  be  assessed 
as  operational  mission  failures.  Thus,  by  operational 
criteria,  RMS  demonstrated  reliability  was  no  more 
than  20  hours  Mean  Time  Between  Operational 
Mission  Failure  during  this  phase  of  testing.  Because 
much  of  the  operating  time  cited  by  the  Navy  was 
not  devoted  to  minehunting  activities,  this  estimate 
should  be  considered  an  upper  bound  for  current  RMS 
operational  reliability  Moreover,  statistical  analysis 
of  all  existing  data  does  not  yet  support  the  Navy’s 
assertions  of  improving  RMS  reliability. 

•  Since  RMS  is  critical  to  achieving  the  Navy’s  sustained  area 
coverage  rate  requirement,  this  Annual  Report  includes  a 
separate  article  on  the  RMS  that  provides  additional  detail. 

LFT&E 

•  LCS  is  not  expected  to  be  survivable  in  high-intensity 
combat  because  the  design  requirements  accept  the  risk  the 
ship  must  be  abandoned  under  circumstances  that  would 
not  require  such  an  action  on  other  surface  combatants. 
Although  the  ship  incorporates  capabilities  to  reduce 
susceptibility  to  attack,  previous  testing  of  analogous 
capabilities  demonstrates  it  cannot  be  assumed  LCS  will  not 
be  hit  in  high-intensity  combat. 

•  During  the  TSST  on  LCS  3,  the  Machinery  Plant  Control 
and  Monitoring  System  (MPCMS)  appeared  to  be 
improperly  controlling  the  ventilation  system  for  the  highest 
of  three  material  conditions  of  damage  control  readiness 
known  as  “Condition  ZEBRA.”  This  could  allow  smoke  to 
spread  through  fire  boundaries.  Pressure  differentials  were 
observed  in  several  spaces  that  made  hatches  and  doors 
difficult  to  operate. 

•  There  is  a  problem  with  the  MPCMS  that  caused  every  fire 
alarm  on  the  ship  to  activate  during  shot  1  of  the  TSST  on 
LCS  3,  even  though  the  fire  was  limited  to  the  01  Level. 
Based  on  discussions  with  system  experts,  this  is  a  known 
problem  with  the  MPCMS. 

SSC  Study 

•  In  February  2014,  the  Secretary  of  Defense  directed  the 
Secretary  of  the  Navy  and  the  Chief  of  Naval  Operations 
to  “Submit  to  me,  in  time  to  inform  the  PB  2016  budget 
deliberations,  alternative  proposals  to  procure  a  capable 
and  lethal  small  surface  combatant,  generally  consistent 
with  the  capabilities  of  a  frigate.”  In  October  2014, 

the  Secretary  of  Defense  requested  DOT&E  provide  an 
independent  assessment  of  the  work  done  by  the  SSC  Task 
Force  established  by  the  Navy  pursuant  to  the  Secretary’s 
direction.  In  response,  DOT&E  provided  a  written 
classified  assessment  report  to  the  Secretary. 

•  In  its  report,  DOT&E  concluded  that  the  Navy’s  SSC  Task 
Force’s  results  indicate,  of  the  alternatives  it  considered, 
the  multi-mission  combat  capabilities  and  survivability 
design  features  of  a  modern  frigate  could  be  provided  only 
by  a  new  ship  design  or  a  major  modification  to  the  LCS 
design  -  the  so-called  large  plug  insertion  developed  by 
the  Task  Force.  While  offering  some  improvements  in 


combat  capability  and  survivability  (primarily  via  reduced 
susceptibility)  relative  to  LCS,  the  minor  modifications  to 
LCS  considered  by  the  Task  Force  and  recommended  by 
the  Navy  Leadership  do  not  satisfy  significant  elements 
of  a  capability  concept  developed  by  the  Task  Force  for 
a  modern  frigate.  (The  Task  Force  developed  a  number 
of  capability  concepts  incorporating  various  mixes  of 
capabilities  consistent  with  a  frigate.  After  consulting 
with  the  Task  Force’s  lead,  DOT&E’s  assessment  used  one 
particular  concept  as  representative  of  a  modern  frigate’s 
capabilities.  Also,  “major  modification  to  LCS”  and  “minor 
modification  to  LCS”  are  the  characterizations  used  by  the 
Task  Force  of  its  alternatives.)  Notwithstanding  potential 
reductions  to  its  susceptibility  relative  to  LCS,  DOT&E’s 
assessment  is  that  minor  modifications  to  LCS  will  not  yield 
a  ship  that  is  significantly  more  survivable  than  LCS. 

•  DOT&E  also  noted  in  its  report  provided  to  the  Secretary 
that  DOT&E’s  assessment  was  based  on  results  that  might 
subsequently  change,  because  the  Task  Force’s  report 
remained  unfinished  at  the  time  of  DOT&E’s  report. 

Recommendations 

•  Status  of  Previous  Recommendations. 

-  The  Navy  partially  addressed  one  FY09  recommendation 
to  develop  an  LFT&E  program  with  the  approval  of  the 
LFT&E  Management  Plan;  however,  the  details  of  the 
surrogate  testing  and  the  lethality  testing  still  need  to  be 
developed. 

-  The  Navy  partially  addressed  the  FY10  recommendations 
to  implement  recommendations  from  DOT&E’s  Combined 
Operational  and  Live  Fire  Early  Fielding  Report. 

Significant  remaining  recommendations  include  enhancing 
seaframe  sensors  and  improving  capability  of  seaframe  and 
SUW  Mission  Package  gun  systems. 

-  With  respect  to  FY 1 1  recommendations  regarding 
AN/AQS-20A  and  ALMDS,  the  Navy  is  adjusting  tactics 
and,  for  the  AN/AQS-20A,  funding  improvements  to 
address  deficiencies.  The  FY11  recommendation  for  the 
Navy  to  continue  to  report  vulnerabilities  during  live  fire 
tests  remains  valid. 

-  For  FY12  recommendations: 

■  The  Navy  partially  addressed  the  recommendations  to 
complete  the  revised  capabilities  document  defining  the 
incremental  approach  to  fielding  mission  packages. 

■  The  Navy  has  released  requirements  letters  for 
Increments  1  and  2  SUW  and  Increment  1  MCM  Mission 
Packages  only;  however,  the  requirements  have  not 
been  codified  in  an  approved  Capabilities  Production 
Document.  The  Navy  published  the  LCS  Platform 
Wholeness  Concept  of  Operations  Revision  D  in 
January  2013. 

■  The  Navy  has  not  published  the  concept  of  employment 
for  all  the  mission  packages,  but  advises  that  initial 
manning  level  studies  have  been  completed.  The  Navy 
has  adjusted  ship  and  mission  package  manning  levels 
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and  is  continuing  studies  to  determine  the  final  manning 
levels. 

■  The  Navy  has  stated  that  gun  reliability  problems 
identified  during  the  Quick  Reaction  Assessment  have 
been  resolved  based  on  limited  testing  conducted  in 
October  2012.  Additional  testing  conducted  aboard 
LCS  3  in  2013  and  2014,  which  was  observed  by 
DOT&E,  indicates  that  the  gun  reliability  has  improved. 

■  The  Navy  conducted  LCS  ship-based  phases  of  the 
planned  operational  assessments  of  the  MH-60S 
Block  2/3  and  ALMDS  and  the  MH-60S  Block  2/3  and 
AMNS  MCM  systems  in  1QFY15. 

■  Throughout  FY 13/14,  the  Navy  focused  on  correction  of 
material  deficiencies  with  seaframe  launch  and  recovery 
systems,  and  manpower  and  training  deficiencies  that 
prevent  safe  shipboard  launch  and  recovery  of  the  RMS, 
and  can  now  launch  and  recover  the  RMMV  with  less 
frequent  damage  to  equipment  in  low  to  moderate  sea 
states. 

-  The  Navy  should  still  address  the  FY13  recommendation  to 
provide  a  Surface-to-Surface  Missile  LFT&E  Management 
Plan  for  DOT&E  approval. 

•  FY  14  Recommendations.  The  Navy  should: 

1 .  Continue  to  address  material  reliability  issues  for  both  ship 
classes. 

2.  Address  the  cybersecurity  vulnerabilities  identified  during 
operational  testing  of  the  Freedom  class.  Conduct  in-depth 


cybersecurity  testing  of  the  Independence  class  as  soon  as 
practicable  and  address  deficiencies. 

3.  Emphasize  live-fire  swarm  engagements  for  future  testing 
of  the  SUW  Mission  Package  to  enhance  confidence  in  the 
probability  of  successful  engagement.  As  the  SUW  Mission 
Package  matures  to  Increments  3  and  4,  focus  testing  on 
more  challenging  threats. 

4.  Investigate  the  use  of  communications  relays  and  other 
solutions  that  might  improve  the  standoff  distance  between 
an  RMMV  and  its  host  ship  to  improve  the  efficiency  of 
LCS  MCM  operations. 

5.  Improve  mission  system  (RMMV,  ALMDS,  AMNS, 

AMCM  mission  kit,  AN/AQS-20A/B)  reliability  to 
facilitate  timely  and  sustained  MCM  operations  without 
excessive  reliance  on  shore -based  support. 

6.  Continue  to  investigate  the  root  cause  of  contact 
management  and  communications  problems  observed 
during  recent  MCM  developmental  testing;  develop 
corrective  actions. 

7.  Develop  corrective  actions  to  eliminate  early  termination 
fiber  optic  communications  losses  observed  in  the  AMNS 
operational  assessment;  fund  and  develop  LCS  near-surface 
mine  neutralization  capability. 

8.  Review  the  ventilation  lineup  during  condition  ZEBRA  to 
determine  if  the  system  is  operating  as  intended. 

9.  Correct  problems  with  the  MPCMS  fire  alarm  system. 
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Mark  XIIA  Mode  5  Identification  Friend  or  Foe  (IFF) 


Executive  Summary 

•  Independent  Mark  XIIA  Mode  5  Identification  Friend  or  Foe 
(IFF)  (referred  to  as  “Mode  5”)  development  efforts  exist  in 
each  U.S.  Military  Service  as  well  as  some  NATO  countries. 
Since  Mode  5  is  not  a  joint  program,  the  Services  are 
separately  developing  IFF  equipment  for  use  on  various  land, 
sea,  and  air  platforms. 

-  Of  these  separate  Service  efforts,  only  the  Navy  has  an 
established  Acquisition  Category  II  program. 

-  The  Army  and  Marine  Corps  are  procuring  Mode  5 
transponders  developed  in  the  Navy  program. 

-  Separately,  the  Army  developed  and  is  fielding  its  own 
Mode  5  Air  Defense  Interrogator. 

-  The  Air  Force  is  developing  its  own  Mode  5  transponders 
and  interrogators. 

•  The  Services  are  designing  and  building  Mode  5  systems  to 
comply  with  NATO  and  DOD  IFF  standards.  DOT&E,  in 
coordination  with  the  Joint  Staff  and  USD(AT&L),  initiated 
oversight  in  2006  because  of  the  concern  that  the  multiple 
programs  and  vendors  required  oversight  to  reduce  risks  in 
achieving  joint  IFF  interoperability. 

•  DOT&E  published  the  Mark  XXIIA  Mode  5  Joint  Operational 
Test  Approach  (JOTA)  2  Interoperability  Assessment  in 

July  2014  based  on  the  June  2013  major  joint  operational  test 
off  the  U.S.  East  Coast  that  focused  on  Mode  5  interoperability 
and  identification  in  a  system-of-systems  (SoS)  context.  This 
two  week  event  included  extensive  participation  by  joint 
Service  and  allied  systems  equipped  with  a  wide  variety  of 
Mode  5  equipment  produced  by  different  U.S.  and  European 
manufacturers. 

-  Test  results  showed  that  the  following  Mode  5 
transponders  and  interrogators  were  interoperable  in  a  joint 
SoS  environment: 

■  Army:  Sentinel;  Air  Traffic  Navigational,  Integration, 
and  Coordination  System;  A/MH-6;  C-12R;  CH-47F; 
UH-60M 

■  Air  Force:  F-15C/D/E,  F- 16  Block  40/50 

■  Marine  Corps:  US-1Y 

■  Navy:  E-2C,  MH-60R,  P-3C 

-  DOT&E  observed  some  deficiencies  during  interrogation 
with  the  Patriot  and  Aegis  systems,  which  were 
documented  in  the  classified  assessment. 

-  Patriot  and  Aegis  will  be  required  to  participate  in 
additional  joint  interoperability  tests. 

-  Similar  future  events  will  evaluate  Mode  5  interoperability 
and  identification  as  other  IFF  systems  in  development  are 
integrated  into  various  Service  platforms. 

System 

•  The  Mark  XIIA  Mode  5  IFF  is  a  cooperative  identification 
system  that  uses  interrogators  and  transponders  on  host 


1  -  Replacement  Card  for  Mode  5  Capability 

2  -  RT-1832  F/A-18  Interrogator 

3  -  APX-123  Navy  Transponder  (All  Platforms) 

4  -  UPX-41C  Shipboard  Interrogator 


platforms  to  send,  receive,  and  process  friendly  identification 
data. 

•  Mode  5  is  a  military-only  identification  mode,  which  modifies 
the  existing  Mark  XII  Mode  4  IFF  (referred  to  as  “Mode  4”) 
system  and  addresses  known  shortcomings  of  the  legacy 
Mode  4  identification  mode.  Mode  5  will  eventually  replace 
Mode  4  and  allows  National  Security  Agency-certified  secure 
encryption  of  interrogations  and  replies.  Primary  Mode  5 
system  features  include: 

-  A  Lethal  Interrogation  format,  which  is  used  by  a 
weapons-capable  platform  prior  to  weapons  release  as  a 
final  attempt  to  get  a  valid  Mode  5  reply  from  the  target, 
even  with  the  target’s  interrogated  Mode  5  transponder 
system  in  standby;  this  is  intended  to  reduce  the  possibility 
of  fratricide. 

-  A  random-reply-delay,  which  prevents  overlapping 
replies  and  provides  better  display  discrimination  for 
closely-spaced  platforms. 

•  Mode  5  offers  more  modern  signal  processing,  compatibility 
with  legacy  Mode  4  systems  and  civilian  air  traffic  control, 
and  secure  and  encrypted  data  exchange  through  the  use  of  a 
new  waveform. 

•  Mode  5  serves  as  a  component  of  the  combat  identification 
process  used  on  ground-based  systems  such  as  the  Army’s 
Patriot  and  Sentinel  missile  systems,  sea-based  systems 
such  as  Aegis-equipped  warships,  and  military  aircraft  to 
include  the  E-3  Sentry  Airborne  Warning  and  Control  System 
(AWACS)  and  E-2  Hawkeye  command  and  control  platforms. 

•  Independent  Mode  5  development  efforts  exist  in  each 
U.S.  Military  Service  as  well  as  some  NATO  countries. 
Although  not  a  joint  program,  the  Services  are  developing 
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equipment  capable  of  employment  on  multiple  Service 
platforms. 

-  Of  the  four  separate  Service  efforts,  only  the  Navy 
had  an  established  Acquisition  Category  II  Program  of 
Record,  with  incorporation  of  Mode  5  capability  through 
platform-specific  Engineering  Change  Proposals. 

-  The  Army  and  Marine  Corps  are  leveraging  the  Navy 
program,  and  the  Air  Force  is  executing  individual 
Engineering  Change  Proposals  to  introduce  Mode  5 
capabilities  into  its  various  platforms. 

Mission 

The  Combatant  Commander  employs  the  Mode  5  system  to 
provide  positive,  secure,  line-of-sight  identification  of  friendly 


platforms  equipped  with  an  IFF  transponder.  In  the  future,  this 
system’s  information  will  be  combined  with  other  cooperative 
and  non-cooperative  combat  identification  techniques  in  order 
to  provide  identification  of  all  platforms — enemy,  neutral,  and 
friendly. 

Major  Contractors 

•  Navy  Transponder  and  Interrogator:  BAE 
Systems  -  Arlington,  Virginia 

•  Air  Force  Transponder  and  Interrogator  and  Army  Air  Defense 
Interrogator:  Raytheon  Systems  -  Waltham,  Massachusetts 

•  Air  Force  E-3  Interrogator:  Telephonies 
Corporation  -  Farmingdale,  New  York 


Activity 

•  The  Navy  conducted  a  JOTA  event  in  3QFY13  off  the 
U.S.  East  Coast  during  the  Joint  Staff  J-6-led  Bold  Quest 
Coalition  Capability  Demonstration  and  Assessment  event, 
which  involved  a  mixture  of  blue  and  red  forces  consisting 
of  a  variety  of  joint  Service  and  allied  aircraft  equipped 
with  interrogators  and  transponders  produced  by  different 
U.S.  and  allied  manufacturers. 

-  Air  warfare  scenarios  were  conducted  under  Navy 
Aegis  warship,  AWACS,  or  ground  controlled  intercept 
control. 

-  During  the  JOTA  event,  U.S.  and  allied  aircraft  flew 
272  of  294  planned  aircraft  sorties.  Representative 
operational  flight  profiles  and  tactics  were  used  during 
the  event. 

•  Future  JOTA  events  are  intended  to  support  the  planned 
FY20  Full  Operational  Capability  declaration. 

•  The  next  JOTA  event  is  expected  in  the  FY17/18  timeframe 
and  is  driven  by  systems  ready  to  test  as  well  as  test  venue 
availability. 

•  DOT&E  published  the  Mark  XIIA  Mode  5  JOTA  2 
Interoperability  Assessment  in  July  2014. 

Assessment 

•  The  Navy  Aegis  Weapon  System  was  initially  unable  to 
come  online  at  the  start  of  its  scheduled  participation  in 
the  joint  interoperability  exercise  due  to  the  fact  that  the 
published  procedures  for  the  initialization  of  the  Mode  5 
interrogator  as  installed  in  the  Aegis  system  were  incorrect. 
Subsequent  to  the  JOTA  2  Interoperability  Assessment,  the 
Navy  developed  and  disseminated  a  procedural  change  to 
the  Aegis  operator  manuals  to  correct  this  issue. 

•  The  3QFY13  JOTA  test  event  addressed  DOD  concerns 
about  joint  interoperability  and  identification  in  an  SoS 
context  for  those  systems  under  test. 

-  Results  revealed  no  new  Mode  5-associated 
deficiencies  and  the  following  Mode  5  transponders 


and  interrogators  were  interoperable  in  a  joint  SoS 
environment: 

■  Army  -  Sentinel;  Air  Traffic  Navigational,  Integration, 
and  Coordination  System;  A/MH-6;  C-12R;  CH-47F; 
UH-60M 

■  Air  Force  -  F-15C/D/E,  F-16  Block  40/50 

■  Marine  Corps-US-lY 

■  Navy  -  E-2C,  MH-60R,  P-3C 

-  Both  Patriot  and  Aegis  need  to  improve  their  ability  to 
fully  employ  the  new  Lethal  Interrogation  capability 
inherent  in  the  Mode  5  system. 

-  Identification  information  from  some  Mode  5-equipped 
command  and  control  systems  could  not  be  directly 
passed  via  Link  16  datalinks  due  to  inconsistent 
implementation  of  the  latest  version  of  the  Military 
Standard  for  dissemination  of  Mode  5  messages  over 
Link  16. 

■  Prior  to  the  JOTA  2  interoperability  exercise,  some 
systems  had  implemented  Military  Standard-60 16E  with 
others  scheduled  to  receive  it  later.  For  those  systems 
that  did  not  feature  6016E  at  the  time  of  the  JOTA  2 
exercise,  a  complex  and  time-consuming  methodology 
was  developed  and  utilized  post-exercise  to  allow 
Mode  5  track  data  to  be  reconstructed  and  used  in  the 
interoperability  and  effectiveness  assessment. 

•  The  Services  are  implementing  new  software  builds  to 
enable  Link  16-equipped  systems  to  disseminate  Mode  5 
identification  information.  Specifically,  Lethal  Interrogation 
capability  with  Patriot  and  Aegis  is  planned  to  be  improved. 

-  The  installed  performance  of  new  software  builds,  as 
well  as  Mode  5  interoperability  with  both  existing  and 
planned  IFF  systems,  is  being  validated  in  combined 
developmental/integration  testing.  These  upgrades  will 
be  incorporated  into  Mode  5-equipped  systems  over  the 
next  several  years. 


206  Mark  XIIA  Mode  5  IFF 


F Y1  4  NAVY  PROGRAMS 


Recommendations 

•  Status  of  Previous  Recommendations.  The  Services  have 
adequately  addressed  previous  recommendations  for  some 
platforms;  however,  response  to  these  FY13  recommendations 
is  ongoing  as  Mode  5  integration  occurs  on  additional 
platforms. 

•  FY 14  Recommendations. 

1 .  In  order  to  ensure  interoperability  between  interrogators, 
transponders,  and  combined  interrogator-transponders, 
Service  program  managers  must  ensure  that  developmental 


and  operational  testing  of  Mode  5  capabilities  and  systems 
address  compatibility  with  both  joint  Service  and  allied  IFF 
systems  as  well  as  interoperability  with  Link  16. 

2.  All  platforms  that  did  not  participate  in  JOTA  2  and  are 
equipped  with  Mode  5  capability  must  participate  in  a 
future  JOTA  equivalent  SoS  interoperability  event. 

3.  Patriot  and  Aegis  must  return  for  a  subsequent  JOTA 
equivalent  SoS  event  to  assess  correction  of  deficiencies 
during  JOTA  2. 
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MH-60R  Multi-Mission  Helicopter 


Executive  Summary 

•  Commander,  Operational  Test  and  Evaluation  Force  (COTF) 
completed  testing  in  FY13  on  corrections  made  to  resolve 
previously  identified  deficiencies  with  AGM-114  HELLFIRE 
missiles  and  the  Multi-spectral  Targeting  System  (MTS). 
DOT&E  issued  a  classified  report  in  April  2014  and  noted  that 
while  some  deficiencies  with  MTS  were  resolved,  it  still  does 
not  adequately  meet  its  tracking  requirement. 

•  COTF  completed  IOT&E  of  MH-60R  with  the  Automatic  Radar 
Periscope  Detection  and  Discrimination  (ARPDD)  upgrade  in 
1QFY14.  In  April  2014,  DOT&E  issued  a  classified  IOT&E 
report  and  concluded  that  ARPDD  was  operationally  effective 
but  performance  was  significantly  affected  by  environmental 
conditions.  DOT&E  did  not  resolve  ARPDD  suitability  due  to 
uncertainty  in  system  reliability  that  requires  further  monitoring. 

•  The  January  2014  FOT&E  for  the  Mk  54  lightweight  torpedo 
with  Block  Upgrade  (BUG)  tactical  software  identified 
shortfalls  with  the  MH-60R  tactics  and  tactical  documentation, 
and  interoperability  problems  with  the  helicopter’s  fire  control 
systems. 

System 

•  The  MH-60R  is  a  ship-based  helicopter  designed  to  operate 
from  cruisers,  destroyers,  frigates,  littoral  combat  ships,  and 
aircraft  carriers. 

•  It  incorporates  dipping  sonar  and  sonobuoy  acoustic  sensors, 
multi-mode  radar,  electronic  warfare  sensors,  a  forward-looking 
infrared  sensor  with  laser  designator,  and  an  advanced  mission 
data  processing  system. 

•  It  employs  Mk  46  and  Mk  54  torpedoes,  HELLFIRE 
air-to-ground  missiles,  2.75-inch  family  of  rockets,  and 
crew-served  mounted  machine  guns. 

•  It  has  a  three-man  crew:  two  pilots  and  one  sensor  operator. 


Mission 

The  Maritime  Component  Commander  employs  the  MH-60R 

from  ships  or  shore  stations  to  accomplish  the  following: 

•  Surface  Warfare,  Under  Sea  Warfare,  Area  Surveillance, 
Combat  Identification,  and  Naval  Surface  Fire  Support 
missions  previously  provided  by  two  different  helicopters 
(SH-60B  and  SH-60F) 

•  Support  missions  such  as  Search  and  Rescue  at-sea  and,  when 
outfitted  with  necessary  armament,  maritime  force  protection 
duties 

Major  Contractors 

•  Sikorsky  Aircraft  Corporation  -  Stratford,  Connecticut 

•  Lockheed  Martin  Mission  System  and 
Sensors  -  Owego,  New  York 


Activity 

•  COTF  executed  FOT&E  on  the  MH-60R  helicopter 
employing  AGM-114  HELLFIRE  missiles  with  the  MTS  from 
4QFY12  to  2QFY13.  This  testing  focused  on  corrections 
made  to  resolve  previously  identified  MTS  deficiencies. 
DOT&E  issued  a  classified  report  on  the  completed  FOT&E 
on  April  28,  2014. 

•  COTF  conducted  ARPDD  testing  between  August  2012 
and  September  2013  that  included  three  different 
locations:  the  Narragansett  Bay  Operating  Area  near 

Cape  Cod,  Massachusetts;  the  Patuxent  River  Operating  Area 
off  the  Maryland  coast;  and  the  Southern  California  Offshore 


Range  off  the  coast  of  San  Diego,  California.  This  testing 
focused  on  determining  ARPDD  capability  to  detect  and 
classify  threat-representative  scope  exposures  across  a  range 
of  distance,  environmental,  and  exposure  duration  conditions. 
DOT&E  issued  a  classified  report  on  completed  IOT&E  of 
MH-60R  with  the  ARPDD  upgrade  on  April  3,  2014. 

•  The  Navy  completed  FOT&E  for  the  Mk  54  lightweight 
torpedo  with  BUG  tactical  software  in  January  2014.  The 
MH-60R  served  as  the  launch  platform  for  14  weapon  drops. 

•  COTF  conducted  all  testing  in  accordance  with  a 
DOT&E-approved  test  plan. 
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Assessment 

•  The  upgraded  MTS  software  showed  some  improved 
performance  compared  to  prior  operational  testing,  but  the 
MTS  still  did  not  adequately  meet  its  requirement  for  tracking. 
The  Navy  conducted  a  demonstration  of  the  Surface  Warfare 
mission  capability  of  the  MH-60R  helicopter  equipped  with 
the  HELLFIRE  missile  and  MTS;  however,  testing  throughout 
the  operational  mission  environment  is  not  complete. 

•  IOT&E  demonstrated  that  ARPDD  was  operationally 
effective,  but  environmental  conditions  significantly 
affected  performance.  Periscope  detection  and  classification 
performance  were  demonstrated  at  ranges  that  should  support 
effective  Anti-Submarine  Warfare  in  low,  sea-clutter  conditions 
that  are  routinely  experienced  in  the  defense  of  a  battlegroup 
in  open-ocean.  Periscope  detection  and  classification  were 
degraded  by  significant  numbers  of  non-periscope  detections 
in  high-clutter  environments  that  may  be  observed  in  littoral 
waters,  an  area  of  interest  due  to  the  prevalence  of  threat 
submarines  in  these  regions.  Overall,  ARPDD  provided 
MH-60R  helicopters  with  a  submarine  identification  capability 
that  exceeds  radar  systems  used  on  other  Anti-Submarine 
Warfare  air  platforms.  As  with  all  radar  systems,  ARPDD 
requires  a  periscope  exposure  and  has  no  detection  capability 
against  submarines  that  remain  submerged  below  periscope 
depth. 

•  During  dedicated  operational  testing,  the  AN/APS-153  radar 
system  with  ARPDD  demonstrated  a  Mean  Time  Between 
Failure  of  60  hours,  well  below  the  fleet  operational  reported 
Mean  Time  Between  Failure  of  nearly  1,000  hours.  The 
cause  of  the  significant  discrepancy  is  unknown  and  did  not 
support  resolution  of  ARPDD  suitability.  However,  the  much 
more  extensive  set  of  fleet  data  is  likely  a  better  indicator  of 
the  radar’s  reliability  than  the  more  limited  set  of  data  from 
ARPDD  test  events. 

•  Operational  testing  of  the  Mk  54  BUG  torpedo  identified 
shortfalls  with  MH-60R  tactics  and  tactical  documentation 


and  interoperability  problems  with  the  helicopter’s  fire  control 
systems.  Navy  testers  discovered  that  MH-60R  crews  could 
not  access  some  weapon  presets.  These  problems  could 
prevent  fleet  operators  from  effectively  employing  the  Mk  54 
BUG  torpedo.  The  Navy  initiated  immediate  actions  to 
address  this  shortfall. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  partially 
addressed  the  FY12  recommendation  to  assess  corrections 
made  to  resolve  previously  identified  MTS  deficiencies  by 
conducting  FOT&E.  The  Navy  has  not  acted  on  the  FY13 
recommendations  to  conduct  comprehensive  live  fire  lethality 
testing  of  the  HELLFIRE  missile  against  a  complete  set 

of  threat-representative  small  boat  targets  and  to  test  the 
Surface  Warfare  mission  capability  of  MH-60R  equipped  with 
HELLFIRE  missiles. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Continue  to  correct  the  remaining  deficiencies  with  the 
MTS  tracker. 

2.  Demonstrate  the  Surface  Warfare  mission  capability  of  the 
MH-60R  helicopter  equipped  with  the  HELLFIRE  missile 
and  MTS  throughout  the  operational  mission  environment 
in  FOT&E  and  LFT&E. 

3.  Conduct  further  evaluation  of  ARPDD  employment  in 
high-clutter,  high-contact  density  littoral  environments. 

4.  Collect  additional  radar  reliability  data,  particularly  during 
periods  of  operations  that  employ  the  ARPDD  mode. 

5.  Address  the  additional  six  recommendations  in  the 
classified  IOT&E  report  on  MH-60R  with  the  ARPDD 
system. 

6.  Investigate  and  correct  interoperability  deficiencies  of  the 
Mk  54  with  MH-60R  weapons  control  systems. 
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MH-60S  Multi-Mission  Combat  Support  Helicopter 


Executive  Summary 

•  Commander,  Operational  Test  and  Evaluation  Force  (COTF) 
completed  testing  in  FY13  on  corrections  made  to  resolve 
previously  identified  deficiencies  with  AGM-114  F1ELLFIRE 
missiles  and  the  Multi-spectral  Targeting  System  (MTS). 
DOT&E  issued  a  classified  report  in  April  2014  and  noted  that 
while  some  deficiencies  with  MTS  were  resolved,  it  still  does 
not  adequately  meet  its  tracking  requirement. 

•  DOT&E  provided  an  analysis  of  the  results  of  the  Quick 
Reaction  Assessment  (QRA)  of  the  LAU-61G/A  Digital 
Rocket  Launcher  using  Advanced  Precision  Kill  Weapon 
System  II  (APKWS)  rockets  in  June  2014.  The  analysis 
showed  that  the  Digital  Rocket  Launcher  with  APKWS 
rockets  provides  additional  Surface  Warfare  capability  to  the 
MH-60S,  but  noted  technical  and  operational  risks  that  should 
be  addressed  for  improved  performance. 

•  COTF  completed  Phase  A  (shore -based  and  training  phase) 
of  the  planned  operational  assessment  of  the  MH-60S 
Block  2  Airborne  Mine  Neutralization  System  (AMNS)  in 
3QFY 14.  Failures  of  the  host  MH-60  aircraft’s  systems  and 
its  associated  Airborne  Mine  Countermeasures  (AMCM) 
kit  severely  limited  AMNS  availability.  Frequent  loss  of 
fiber-optic  communications  between  the  aircraft  and  the 
neutralizer  was  the  primary  cause  of  unsuccessful  attack  runs. 
Both  problems  increase  the  time  needed  to  conduct  Littoral 
Combat  Ship  (LCS)-based  AMNS  operations  and  reduce  the 
ship’s  sustained  area  coverage  rate.  COTF  completed  an 
operational  assessment  of  the  Airborne  Lase  Mine  Detection 
System  (ALMDS)  and  AMNS  by  conducting  Phase  B 
(LCS)-based  phase)  of  the  test  in  1QFY15  aboard  LCS  2 
during  Increment  1  MCM  mission  package  developmental 
testing.  DOT&E  assessed  the  detection  and  localization 
capabilities  provided  by  ALMDS  and  the  identification  and 
neutralization  capabilities  provided  by  AMNS  to  be  consistent 
with  the  findings  from  earlier  shore-based  operational 
assessments  of  the  systems.  Neither  system  demonstrated 
the  performance  necessary  for  an  LCS  equipped  with  an 
Increment  1  MCM  mission  package  to  meet  its  required 
sustained  area  coverage  rate. 

System 

•  The  MH-60S  is  a  helicopter  modified  into  three  variants 
(blocks)  from  the  Army  UH-60L  Blackhawk  for  operation  in 
the  shipboard/maritime  environment. 

•  The  blocks  share  common  cockpit  avionics  and  flight 
instrumentation  with  the  MH-60R. 

•  Installed  systems  differ  by  block  based  on  mission: 

-  Block  1,  Fleet  Logistics  -  precision  navigation  and 
communications,  maximum  cargo  or  passenger  capacity. 

-  Block  2  A/B,  AMCM  System  -  AMCM  system  operator 
workstation;  a  tether/towing  system  and  the  two  MCM 


systems  currently  under  development;  ALMDS  for 
detection  and  classification  of  near-surface  mines;  and  the 
AMNS  for  neutralization  of  in- volume  and  bottom  mines. 
In  FY12,  the  Chief  of  Naval  Operations  concluded  that  the 
MH-60S  helicopter  is  significantly  underpowered  for  the 
safe  performance  of  the  AMCM  tow  mission  and  provides 
limited  tactical  utility  relative  to  the  risk  to  aircrew,  and 
cancelled  that  MH-60S  mission.  The  decision  to  cancel 
the  AMCM  tow  mission  affects  employment  of  both 
the  AQS-20A  sonar  and  Organic  Airborne  and  Surface 
Influence  Sweep.  Both  systems  are  no  longer  being 
developed  for  use  in  the  AMCM  system.  Any  Block  2  or 
subsequent  aircraft  (e.g.,  Block  3  A/B  aircraft)  can  be  an 
AMCM  aircraft. 

-  Block  3  A,  Armed  Helicopter  -  20  mm  Gun  System, 
forward-looking  infrared  with  laser  designator, 
crew-served  side  machine  guns,  dual-sided  HELLFIRE 
air-to-ground  missiles,  and  defensive  electronic 
countermeasures. 

-  Block  3B,  Armed  Helicopter  -  Block  3A  with  addition  of 
tactical  datalink  (Link  16)  and  tie-beam  to  the  External 
Weapon  System  Pylon  to  support  the  required  weight  of 
the  loaded  LAU-61s. 

Mission 

The  Maritime  Component  Commander  can  employ  variants  of 

MH-60S  from  ships  or  shore  stations  to  accomplish  the  following 

missions: 

•  Block  1  -  Vertical  replenishment,  internal  cargo  and  personnel 
transport,  medical  evacuation,  Search  and  Rescue,  and  Aircraft 
Carrier  Plane  Guard 

•  Block  2  -  Detection,  classification,  identification  and/or 
neutralization  of  sea  mines,  depending  on  which  AMCM 
systems  are  employed  on  the  aircraft 
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•  Block  3  -  Combat  Search  and  Rescue,  Surface  Warfare, 
Aircraft  Carrier  Plane  Guard,  Maritime  Interdiction 
Operations,  and  Special  Warfare  Support 

Major  Contractors 

•  Sikorsky  Aircraft  Corporation  -  Stratford,  Connecticut 

•  Lockheed  Martin  Mission  System  and 
Sensors  -  Owego,  New  York 


•  Raytheon  Company,  Integrated  Defense 
Systems  -  Tewksbury,  Massachusetts 

•  Northrop  Grumman  -  Melbourne,  Florida 


Activity 

•  COTF  executed  FOT&E  on  the  MH-60S  helicopter  equipped 
with  AGM-114  HELLFIRE  missiles  and  the  MTS  from 
4QFY12  to  2QFY13.  This  testing  focused  on  corrections 
made  to  resolve  previously  identified  MTS  deficiencies. 

•  DOT&E  issued  a  classified  report  on  completed  FOT&E  of 
MH-60R/S  helicopters  equipped  with  AGM-114  HELLFIRE 
missiles  and  the  MTS  on  April  28,  2014. 

•  COTF  conducted  a  QRA  of  MH-60S  with  the  LAU-61G/A 
Digital  Rocket  Launcher  armed  with  APKWS  II  rockets  in 
1QFY14.  DOT&E  issued  the  QRA  report  in  June  2014. 

•  COTF  completed  Phase  A  (shore -based  and  training  phase)  of 
the  planned  operational  assessment  of  the  MH-60S  Block  2 
AMNS  in  3QFY14.  The  test  included  the  use  of  explosive 
neutralizers  and  inert  training  neutralizers  against  inert  moored 
and  bottom  mine  targets,  in  the  lower  current  environment 
only. 

•  COTF  completed  an  operational  assessment  of  the  ALMDS 
and  AMNS  by  conducting  Phase  B  (LCS-based  phase)  of  the 
test  in  1QFY15  aboard  USS  Independence  (LCS  2)  during 
Increment  1  MCM  mission  package  developmental  testing. 

The  test  examined  the  shipboard  compatibility  of  the  MH-60S 
helicopter  and  the  ALMDS  and  the  AMNS,  and  other  aspects 
of  operational  suitability  necessary  to  support  and  sustain  LCS 
MCM  effectiveness. 

•  COTF  conducted  all  testing  in  accordance  with  a 
DOT&E-approved  test  plan. 

Assessment 

•  The  upgraded  MTS  software  showed  some  improved 
performance  compared  to  prior  operational  testing,  but  the 
MTS  still  did  not  adequately  meet  its  requirement  for  tracking. 
Additionally,  the  Surface  Warfare  mission  capability  of  the 
MH-60S  helicopter  equipped  with  MTS  and  the  HELLFIRE 
missile  was  not  tested  throughout  the  operational  mission 
environment.  MTS  tracking  risks  should  be  addressed  as  soon 
as  possible. 

•  The  QRA  of  the  MH-60S  equipped  with  the  Digital  Rocket 
Launcher  and  APKWS  rockets  demonstrated  additional 
Surface  Warfare  capability  for  the  MH-60S,  but  identified 
technical  and  operational  risks  that  should  be  addressed  for 
improved  mission  performance. 

•  During  the  shore-based  operational  assessment  of  the 
AMNS  in  FY14,  AMNS  was  unable  to  achieve  the  Navy’s 


requirement  for  mine  neutralization  success  except  under 
limited  conditions  not  generally  expected  during  combat. 
Failures  of  the  host  MH-60S  aircraft’s  systems  and  its 
associated  AMCM  Mission  Kit  limited  AMNS  mission 
availability.  Frequent  loss  of  fiber-optic  communications 
between  the  aircraft  and  the  neutralizer  was  the  primary  cause 
of  unsuccessful  attack  runs.  Although  the  Navy  attributed 
the  failures  to  the  bottom  composition  (sand  and  shells),  the 
root  cause  of  these  failures  has  not  yet  been  determined,  and 
the  bottom  compositions  used  in  testing  are  representative  of 
realistic  operating  areas.  Both  problems  negatively  affect  the 
timeliness  of  LCS-based  AMNS  operations  and  will  likely 
reduce  the  ship’s  sustained  area  coverage  rate. 

•  The  results  of  the  Navy’s  1QFY15  operational  assessment  of 
the  ALMDS  and  AMNS  were  consistent  with  the  findings  from 
earlier  shore-based  operational  assessments  of  the  systems: 

-  DOT&E  assessed  the  MH-60S  helicopter  equipped 
with  the  AMNS  and  determined  the  identification  and 
neutralization  capabilities  were  not  demonstrated  for  an 
LCS  equipped  with  an  Increment  1  MCM  mission  package 
to  meet  its  required  sustained  area  coverage  rate.  No 

data  were  collected  to  assess  AMNS  performance  in  the 
medium  or  higher  current  regimes. 

-  DOT&E  assessed  the  MH-60S  helicopter  equipped  with 
the  ALMDS  and  determined  the  detection  and  localization 
capabilities  were  not  demonstrated  for  an  LCS  equipped 
with  an  Increment  1  MCM  mission  package  to  meet 

its  required  sustained  area  coverage  rate.  Shore -based 
testing  of  the  ALMDS  completed  in  4QFY12  showed  that 
the  system  does  not  meet  Navy  detection  requirements. 
ALMDS  generates  a  large  number  of  false  classifications 
(objects  erroneously  classified  as  mine-like).  Unless 
eliminated  from  the  contact  list,  these  false  classifications 
require  identification  and  neutralization  effort,  result 
in  the  expenditure  of  limited  neutralizer  assets,  and 
drastically  reduce  the  LCS  sustained  area  coverage  rate. 

As  an  alternative,  the  Navy  has  implemented  tactics 
that  require  multiple  search  passes  over  the  same  area  to 
minimize  the  number  of  false  classifications  passed  on  for 
identification/  neutralization.  Although  multiple  passes 
also  reduce  the  LCS  sustained  area  coverage  rate  relative  to 
single  pass  systems,  Navy  modeling  suggests  this  approach 
is  less  detrimental  to  MCM  timelines.  Whether  LCS  can 
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meet  the  already-reduced  low  area  clearance  requirement 
for  the  Increment  1  mission  package  remains  in  question. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  still  needs 
to  address  the  FY 1 1  recommendation  to  investigate  solutions 
and  correct  the  ALMDS  False  Classification  Density  and 
reliability  deficiencies  prior  to  IOT&E.  The  Navy  has  partially 
addressed  the  FY12  recommendation  to  assess  corrections 
made  to  resolve  previously  identified  MTS  deficiencies  by 
conducting  FOT&E.  The  Navy  has  not  acted  on  the  FY13 
recommendations  to: 

1.  Complete  comprehensive  survivability  studies  for  MF1-60S 
employing  the  20  mm  Gun  System  and  the  Unguided 
Rocket  Launcher. 

2.  Conduct  comprehensive  live  fire  lethality  testing 
of  the  HELLFIRE  missile  against  a  complete  set  of 
threat- representative  small  boat  targets. 

3.  Correct  the  tracking  deficiencies  in  the  MTS  and  conduct 
appropriate  FOT&E  in  order  to  satisfactorily  resolve  the 
Surface  Warfare  Critical  Operational  Issue. 

4.  Complete  comprehensive  IOT&E  on  the  2.75”  Unguided 
Rocket  and  APKWS  II  to  resolve  the  Surface  Warfare 
Critical  Operational  Issue  not  resolved  in  limited 
assessments  of  system  performance  provided  in  QRAs 
against  small  boat  threats. 


•  FY14  Recommendations.  The  Navy  should: 

1 .  Continue  to  correct  the  deficiencies  with  the  MTS  tracker 
identified  in  testing. 

2.  Test  the  Surface  Warfare  mission  capability  of  MH-60S 
Helicopter  equipped  with  MTS  and  the  HELLFIRE  missile 
throughout  the  operational  mission  environment  in  FOT&E 
and  LFT&E. 

3.  Complete  vulnerability  studies  for  MH-60S  employing 
the  LAU-61G/A  Digital  Rocket  Launcher  armed  with 
APKWS  II  rockets. 

4.  Conduct  comprehensive  lethality  testing  of  the  LAU-61G/A 
Digital  Rocket  Launcher  armed  with  APKWS  II  rockets 
against  a  complete  set  of  threat-representative  small  boat 
targets. 

5.  Correct  AMCM  mission  kit  reliability  issues  that  limit 
AMNS  mission  availability  identified  during  the  operational 
assessment. 

6.  Develop  corrective  actions  to  eliminate  early  termination 
fiber-optic  communications  losses  observed  in  the  AMNS 
operational  assessment. 

7.  Conduct  AMNS  medium  current  testing  from  MH-60S. 

This  testing  must  be  completed  prior  to  the  completion  of 
IOT&E. 
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Mine  Resistant  Ambush  Protected  (MRAP) 
Family  of  Vehicles  (FoV)  -  Marine  Corps 


Executive  Summary 

•  The  U.S.  Marine  Corps  (USMC)  will  retain  2,467  Mine 
Resistant  Ambush  Protected  (MRAP)  vehicles  in  its  enduring 
fleet  that  includes  Cougar  Category  (CAT)  I,  Cougar  CAT  II, 
Buffalo  CAT  III,  and  MRAP  All-Terrain  Vehicle  (M-ATV) 
variants. 

•  Early  live  fire  testing  conducted  in  4QFY14  suggests  that  the 
Cougar  CAT  II A 1  variant  with  seat  survivability  upgrades  will 
meet  its  required  level  of  performance;  however,  additional 
planned  live  fire  testing  remains  to  be  executed  in  FY15. 

•  The  Cougar  retrofitted  with  a  remote  weapons  station  for  use 
by  the  Air  Force  provides  protection  from  underbody  blasts. 

System 

•  The  MRAP  Family  of  Vehicles  (FoV)  is  designed  to  provide 
increased  crew  protection  and  vehicle  survivability  against 
current  battlefield  threats,  such  as  IEDs,  mines,  small  arms  fire 
(SAF),  rocket-propelled  grenades  (RPGs),  and  Explosively 
Formed  Penetrators  (EFPs).  The  MRAPs  are  employed 

by  units  in  current  combat  operations  in  the  execution  of 
missions  previously  accomplished  with  the  High  Mobility 
Multi-purpose  Wheeled  Vehicle. 

•  In  FY14,  the  MRAP  Joint  Program  Office  (JPO)  dissolved 
and  the  Army  and  the  Marine  Corps  became  the  lead  Services 
responsible  for  their  respective  MRAP  variants.  In  2013,  the 
Marine  Corps  defined  its  enduring  MRAP  fleet,  which  it  will 
retain  after  transitioning  from  a  JPO  to  a  USMC-led  program 
manager.  Currently,  2,467  MRAP  vehicles  will  be  retained 
in  the  USMC  enduring  fleet:  CAT  1 A1  (1,337  vehicles); 

CAT  I A1  tube-launched,  optically-tracked,  wireless-guided 
(TOW®)  weapons  system  (59);  CAT  II A1  (300);  Cougar 
A2  Ambulance  (19);  M-ATV  with  Underbelly  Improvement 
Kit  (704);  Cougar  CAT  II  Mounted  Communications  Emitter 
Sensing  Attack  System  (10),  and  Buffalo  CAT  III  (38).  In 
addition,  the  USMC  will  remain  the  Primary  Inventory 
Control  Activity  for  all  Cougar  platforms,  including  those 
vehicles  divested  to  the  Navy  and  Air  Force. 

•  This  report  covers  testing  of  two  USMC  Cougar  CAT  II A 1 
variants:  one  with  seat  survivability  upgrades  (SSU)  and  one 


Cougar  Category  (CAT)  II A1 


with  a  remote  weapons  station  (RWS)  that  is  intended  for  use 
by  the  Air  Force. 

-  The  SSU  is  primarily  a  redesign  of  the  rear  crew 
compartment  of  the  Cougar,  focusing  on  improved  seating 
for  survivability,  safety,  and  human  factors  integration,  and 
can  carry  10  Marines  and  1  gunner. 

-  The  Air  Force  RWS  Cougar  has  a  Common 
Remotely-Operated  Weapons  Station  II  mounted  to  its 
roof,  and  will  carry  a  crew  of  three  (driver,  assistant  driver, 
and  RWS  gunner). 

Mission 

Marines  will  operate  the  MRAP  Cougar  CAT  II A 1  vehicle  to 
conduct  mounted  patrols,  reconnaissance,  communications, 
and  command  and  control  missions  in  a  threat  environment. 
Explosive  Ordnance  teams  will  rely  on  the  Air  Force  RWS 
Cougar  for  support  in  a  wide  range  of  missions  and  environments 
both  in  the  U.S.  and  while  deployed. 

Major  Contractor 

General  Dynamics  Land  Systems  -  Ladson,  South  Carolina 


Activity 

Cougar  CAT  II A1  with  SSU 

•  The  Marine  Corps  plans  to  retrofit  a  total  of  303  USMC, 

41  Air  Force,  and  14  Navy  CAT  II A1  vehicles  with  the 
SSU  upgrades  that  are  designed  to  achieve  Capabilities 
Production  Document  1.1  objective-level  protection  against 
underbody  and  under-wheel  blast  mines. 


Live  fire  testing  of  the  CAT  II A1  Cougar  with  SSU 
commenced  in  FY14  with  an  underbody  blast  event,  crew 
Automatic  Fire  Extinguishing  System  testing,  and  ballistic 
exploitation  testing.  Additional  planned  blast  testing  will 
continue  into  FY15.  This  seat  upgrade  program  is  a  result 
of  the  2010  DOT&E  report  on  the  original  MRAP  Cougar 
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vehicles  and  a  2011  DOT&E  report  on  the  Cougar  vehicles 
retrofitted  with  an  Independent  Suspension  System.  Both 
reports  emphasized  the  need  for  the  then  JPO  to  install 
improved  energy-attenuating  seats  across  the  Cougar  fleet. 

Air  Force  RWS  Cougar 

•  The  Air  Force  will  integrate  the  RWS  upgrade  into  the 
following  Cougar  four  variants:  CAT  1 A1  (9  vehicles), 
CAT  II A1  (39  vehicles),  CAT  I A2  (15  vehicles),  and 
CAT  II A2  (19  vehicles). 

•  The  Marines  executed  a  single  underbody  blast  test  of  a 
CAT  II A1  Cougar  in  the  RWS  configuration.  The  CAT  II 
AI  Cougar  comprises  39  of  the  82  total  vehicles  to  be 
retrofitted,  and  configuration  of  the  RWS  integration  across 
all  four  variants  is  sufficiently  similar  for  the  results  of  that 
test  to  apply  to  all  four  variants.  This  test  focused  on  the 
area  of  significant  modification  for  this  platform,  which  was 
under  the  new  RWS  gunner  seat. 

•  All  testing  was  conducted  in  accordance  with  a 
DOT&E-approved  test  plan. 

Assessment 

Cougar  CAT  II  Al  with  SSU 

•  Early  live  fire  testing  indicates  that  the  Cougar  CAT  II 
Al  with  SSU  meets  its  contract  specifications.  However, 


additional  planned  live  fire  testing  remains  to  determine 
the  level  of  protection  provided  to  occupants  of  the  vehicle. 
The  results  from  the  legacy  Cougar  live  fire  test  program 
(as  found  in  the  2010  DOT&E  report  on  the  original  MRAP 
Cougar  vehicles)  relative  to  other  tested  threats  such  as 
IEDs,  indirect  fire,  SAF,  RPGs,  and  EFPs  are  applicable  to 
the  Cougar  CAT  II  Al  with  SSU. 

Air  Force  RWS  Cougar 

•  The  Air  Force  RWS  Cougar  provides  protection  to  its 

occupants  from  underbody  blasts  at  the  location  tested.  The 
results  from  the  legacy  Cougar  live  fire  test  program  (as 
found  in  the  2010  DOT&E  report  on  the  original  MRAP 
Cougar  vehicles)  relative  to  other  tested  threats  such  as 
IEDs,  indirect  fire,  SAF,  RPGs,  and  EFPs  are  applicable  to 
the  Air  Force  Cougar  with  RWS. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  program  is 
making  progress  implementing  the  previous  recommendations 
regarding  upgrading  the  seats  in  the  Cougar  CAT  11 A 1 
vehicles.  There  was  no  live  fire  or  operational  testing 
conducted  on  the  M-ATV  in  FY14;  therefore,  none  of  the 
FY13  recommendations  apply  to  the  vehicles  tested  in  FY14. 

•  FY 14  Recommendations.  None. 
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Mk  54  Lightweight  Torpedo 


Executive  Summary 

•  The  Navy  and  DOT&E  completed  OT&E  for  the  Mk  54 
lightweight  torpedo  with  Block  Upgrade  (BUG)  tactical 
software  in  January  2014. 

-  OT&E  started  after  the  Navy  fielded  the  Mk  54  (BUG)  in 
January  2012  in  response  to  an  urgent  operational  need. 
During  operational  testing  that  included  fleet  training,  the 
Navy  shot  weapons  from  surface  ships,  fixed-wing  aircraft, 
and  helicopters  against  U.S.  attack  submarine  and  static 
submarine  surrogate  targets. 

-  DOT&E  issued  a  classified  OT&E  report  on  the  Mk  54 
(BUG)  torpedo  in  September  2014.  DOT&E  assessed 
the  Mk  54  (BUG)  torpedo  as  not  operationally  effective. 
During  operationally  challenging  and  realistic  scenarios, 
the  Mk  54  (BUG)  demonstrated  below  threshold 
performance  and  exhibited  many  of  the  same  failure 
mechanisms  observed  during  the  FY04  IOT&E. 

•  The  Mk  54  torpedo  is  operationally  suitable  and  meets 
the  same  reliability  and  availability  requirements  as  the 
baseline  torpedo.  However,  operational  testing  identified 
shortfalls  with  the  employing  platforms’  tactics  and  tactical 
documentation,  and  interoperability  problems  with  some 
platform  fire  control  systems. 

•  The  Navy  continued  development  of  hardware  and  software 
updates  to  the  Mk  54.  The  new  version  is  designated  the 
Mk  54  Mod  1  torpedo  and  is  scheduled  to  begin  OT&E  in 
FY17. 

•  DOT&E  is  participating  in  the  Navy’s  Torpedo  Target  Strategy 
Working  Group  to  identify  and  develop  test  target  surrogates 
for  the  Mk  54.  The  Navy  proposed  a  short-term  strategy  that 
utilizes  three  separate  targets,  each  appropriate  for  specific 
limited  scenarios.  The  Navy  does  not  have  an  adequate 
long-term  strategy.  Currently,  the  strategy  is  not  fully  funded. 

System 

•  The  Mk  54  lightweight  torpedo  is  the  primary 
Anti-Submarine  Warfare  (ASW)  weapon  used  by  U.S.  surface 
ships,  fixed-wing  aircraft,  and  helicopters. 

•  The  Mk  54  combines  the  advanced  sonar  transceiver  of  the 
Mk  50  torpedo  with  the  legacy  warhead  and  propulsion  system 
of  the  older  Mk  46.  Mk  46  and  Mk  50  torpedoes  can  be 
converted  to  an  Mk  54  via  an  upgrade  kit. 

•  The  Mk  54  sonar  processing  uses  an  expandable,  open 
architecture  system.  It  combines  algorithms  from  the  Mk  50 
and  Mk  48  torpedo  programs  with  the  latest  commercial 
off-the-shelf  technology. 

•  The  Navy  designed  the  Mk  54  to  operate  in  shallow-water 
environments  and  in  the  presence  of  countermeasures. 

•  The  Navy  has  designated  the  Mk  54  torpedo  to  replace  the 

Mk  46  torpedo  as  the  payload  section  for  the  Vertical  Launched 
Anti-Submarine  Rocket  for  rapid  employment  by  surface  ships. 


•  The  High- Altitude  Anti-Submarine  Warfare  Weapons 
Capability  (HAAWC)  program  will  provide  an  adapter  kit  to 
permit  long-range,  high-altitude,  GPS-guided  deployment  of 
the  Mk  54  by  a  P-8A  Multi-mission  Maritime  Aircraft. 

•  The  Mk  54  BUG  is  a  software  upgrade  to  the  Mk  54  baseline 
torpedo  designed  to  correct  deficiencies  identified  during  the 
2004  Mk  54  IOT&E. 

•  The  Mk  54  must  be  interoperable  and  compatible  with  the 
analog  or  digital  combat  control  systems  and  software  variants 
installed  on  all  ASW  fixed-wing  and  helicopter  aircraft,  and 
on  the  surface  ship  combat  control  system  variants  used  for 
torpedo  tube  or  ASW  rocket-launched  torpedoes. 

Mission 

Navy  surface  ships  and  aircraft  employ  the  Mk  54  torpedo  as 

their  primary  anti-submarine  weapon: 

•  For  offensive  purposes,  when  deployed  by  ASW  aircraft  and 
helicopters 

•  For  defensive  purposes,  when  deployed  by  surface  ships 

•  In  both  deep-water  open  ocean  and  shallow-water  littoral 
environments 

•  Against  fast,  deep-diving  nuclear  submarines  and  slow 
moving,  quiet,  diesel-electric  submarines 

Major  Contractors 

•  Raytheon  Integrated  Defense 
Systems  -  Tewksbury,  Massachusetts 

•  Progeny  Systems  Corporation  -  Manassas,  Virginia 

•  Boeing  Company  -  St.  Charles,  Missouri 

•  Northrop  Grumman  -  Annapolis,  Maryland 
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Activity 

•  Operational  testing  of  the  Mk  54  (BUG)  torpedo  commenced 
in  FY12,  following  the  early  fielding  of  the  software  upgrade 
in  January  2012  to  address  a  Fifth  Fleet  Urgent  Operational 
Need  threat.  During  operational  testing,  the  Navy  shot 

77  weapons  from  surface  ships,  fixed-wing  aircraft,  and 
helicopters  against  U.S.  attack  submarine  and  static  submarine 
surrogate  targets.  The  Navy  and  DOT&E  completed  OT&E 
for  the  Mk  54  lightweight  torpedo  with  BUG  tactical  software 
in  January  2014  in  accordance  with  a  DOT&E-approved  test 
plan. 

•  DOT&E  issued  a  classified  OT&E  report  on  the  Mk  54  (BUG) 
torpedo  in  September  2014. 

•  The  Program  Executive  Officer  (Submarines)  recommended 
the  unrestricted  fielding  of  the  Mk  54  (BUG)  in  June  2014. 

•  During  FY14,  the  Navy  continued  development  of  hardware 
and  software  updates  to  the  Mk  54.  The  new  version  is 
designated  the  Mk  54  Mod  1  torpedo  and  is  scheduled  to  begin 
OT&E  in  FY17.  The  Navy  is  developing  new  Mk  54  Mod  1 
torpedo  front  end  processors  and  transducers  and  new  tactical 
software  to  address  the  performance  shortfalls  identified  with 
the  Mk  54  (BUG).  The  Navy  plans  to  approve  a  new  set 

of  Mk  54  Mod  1  requirements  documents  and  the  Test  and 
Evaluation  Master  Plan  (TEMP)  in  FY15. 

•  DOT&E  is  participating  in  the  Navy’s  Torpedo  Target  Strategy 
Working  Group  to  identify  and  develop  test  target  surrogates 
for  the  Mk  54.  The  Navy  proposed  a  short-term  strategy  that 
utilizes  three  separate  targets,  each  appropriate  for  specific 
limited  scenarios.  The  Navy  does  not  have  an  adequate 
long-term  strategy.  Currently,  the  strategy  is  not  fully  funded. 

•  DOT&E  is  participating  in  working  groups  and  funding  (as 
a  short-term  resource  enhancement  project)  an  update  to  the 
Weapons  Assessment  Facility  (WAF)  Hardware-in-the-Loop 
model  and  simulation  located  at  the  Naval  Undersea  Warfare 
Center  in  Newport,  Rhode  Island.  The  project  is  intended 

to  improve  the  WAF  for  developing  and  testing  torpedoes 
by  improving  the  modeling  of  the  ocean  environment  and 
improving  target  models. 

Assessment 

•  DOT&E  assessed  that  the  Mk  54  torpedo  is  not  operationally 
effective  as  an  offensive  ASW  weapon.  During  operationally 
challenging  and  realistic  scenarios,  the  Mk  54  (BUG) 
demonstrated  below  threshold  performance  and  exhibited 
many  of  the  same  failure  mechanisms  observed  during  the 
IOT&E.  Torpedo  mission  kill  performance  against  targets 
employing  the  operationally  realistic  evasion  tactics  was 
below  requirement  thresholds.  Performance  is  further 
degraded  when  considering  crew  performance  for  targeting 
and  employing  the  Mk  54  and  the  Navy’s  assessment  of 
warhead  performance  in  operationally  realistic  scenarios. 

•  The  Mk  54  torpedo  is  operationally  suitable  and  meets 
the  same  reliability  and  availability  requirements  as  the 
baseline  torpedo.  However,  operational  testing  identified 
shortfalls  with  the  employing  platforms’  tactics  and  tactical 
documentation,  and  interoperability  problems  with  some 


platform  fire  control  systems.  During  operational  testing 
of  the  Mk  54  BUG  torpedo,  Navy  testers  discovered  that 
crews  could  not  access  some  weapon  presets  on  some  of  the 
intended  launch  platforms.  These  problems  could  prevent  fleet 
operators  from  effectively  employing  the  Mk  54  BUG  torpedo. 
The  Navy  initiated  immediate  actions  to  address  this  shortfall. 

•  Some  operational  realistic  scenarios  have  not  been  assessed 
due  to  the  unavailability  of  target  surrogates  and  the  Navy’s 
safety  regulations  for  shooting  against  manned  submarine 
targets.  Limited  testing  of  these  regions  of  the  torpedo’s 
operational  profile  confirmed  previous  assumptions  were 
not  valid  and  that  future  dedicated  testing  is  required  with 
target  surrogates  that  are  capable  of  being  hit  by  the  Mk  54 
(set-to-hit  testing). 

•  Additional  information  and  a  detailed  assessment  are  available 
in  DOT&E’s  OT&E  classified  report  on  the  Mk  54  (BUG) 
lightweight  torpedo  dated  September  2014. 

•  The  Navy  proposed  some  unmanned  target  surrogates  that 
could  be  used  to  assess  the  performance  of  the  Mk  54  during 
the  end  of  the  torpedo’s  run.  The  proposals  include  an 
unmanned  static  target,  a  mobile  device  towing  electronic 
target  signal  repeater  (unfunded),  and  a  mobile  small 
submarine  surrogate  intended  for  training  (developed  and 
built  but  unfunded  for  validation  or  operations).  However,  the 
Navy  has  not  provided  sufficient  evidence  to  indicate  these 
will  permit  realistic  engagements. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  following  previous 
recommendations  remain  outstanding.  The  Navy  still  needs 
to: 

1.  Conduct  mobile  target  set-to-hit  testing.  The  Navy 
completed  an  initial  terminal  homing  assessment  against 
the  set-to-hit  Steel  SSK  static  target  surrogate;  however, 
the  Navy  deferred  the  mobile  testing  due  to  the  lack  of  a 
suitable  target  surrogate. 

2.  Continue  to  develop  an  LFT&E  strategy  that  includes  the 
firing  of  the  Mk  54  against  appropriate  targets.  The  Navy 
plans  to  conduct  this  testing  with  the  Mk  54  Mod  1  torpedo 
upgrade. 

3.  Fund  an  operationally  realistic  mobile  set-to-hit  target  to 
complete  the  terminal  homing  testing  of  the  Mk  54  torpedo. 
The  Navy  continues  to  investigate  possible  surrogates; 
however,  the  proposals  are  unfunded. 

4.  Propose  alternatives  to  minimize  or  eliminate  the  test  and 
safety  limitations  that  minimize  operational  realism  in 
Mk  54  testing. 

5.  Complete  development  of  the  Mk  54  Mod  1  requirements 
and  TEMP.  This  TEMP  should  include  all  the  necessary 
resources  or  plans  to  develop  the  necessary  resources, 
including  target  and  range  needs,  to  complete  the  remaining 
testing. 

6.  Pursue  development  of  an  evasive  mobile  set-to-hit  target 
and  threat  representative  countermeasures  to  support 
operationally  realistic  development  and  testing  of  the 
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Mk  54  Mod  1  torpedo.  The  targets  identified  by  the  Navy’s 
Torpedo  Target  Strategy  Working  Group  will  support  a 
limited  portion  of  the  required  Mk  54  development  and 
testing. 

7.  Institute  processes  to  verify  the  incremental  upgrades  to  the 
Mk  54  are  interoperable  with  the  variety  of  combat  systems 
on  surface  ship,  aircraft,  and  helicopter  platforms  and  that 
tactics  and  documentation  are  updated. 

•  FY 14  Recommendations.  The  Navy  should  evaluate  and 
incorporate  the  11  recommendations  in  DOT&E’s  Mk  54 
(BUG)  OT&E  report  to  improve  the  effectiveness  of  the 
Mk  54.  Significant  unclassified  recommendations  include: 

1 .  Improve  the  target  detection  localization  and  track 
performance  of  ship  and  aircraft  crews  that  employ  the 


Mk  54.  While  improving  the  sensor  system  capability  on 
ships  and  aircraft  is  a  longer  range  goal,  updating  the  Mk  54 
employment  tactics,  training,  and  documentation  could 
immediately  improve  overall  crew  proficiency  and  ASW 
effectiveness. 

2.  Improve  the  Mk  54’s  effective  target  search  and  detection 
capability.  The  Mk  54  should  be  able  to  effectively  search 
the  area  defined  by  typical  fire  control  solution  accuracy 
and  crew  employment  and  placement  errors. 

3.  Simplify  the  variability  of  Mk  54  employment  options 
and  required  water  entry  points  in  existing  tactical 
documentation. 
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Mobile  Landing  Platform  (MLP)  Core  Capability  Set 
(CCS)  and  Afloat  Forward  Staging  Base  (AFSB) 


Executive  Summary 

•  In  June  2014,  the  Navy  commenced  Post  Delivery  Test  and 
Trials  (PDT&T)  on  USNS  Montford  Point  (MLP-1)  Mobile 
Landing  Platform  (MLP)  Core  Capability  Set  (CCS).  IOT&E 
took  place  August  25  through  November  4,  2014.  Initial 
results  indicate  the  following: 

-  MLP  (CCS)  is  capable  of  transiting  the  required 
9,500  nautical  miles  at  15  knots  unrefueled. 

-  MLP  (CCS)  can  land  and  launch  Landing  Craft  Air 
Cushion  (LCAC)  vehicles  through  Sea  State  3. 

-  MLP  (CCS)  can  operate  skin-to-skin,  to  include 
vehicle  transfer  through  Sea  State  3,  with  both  the 
USNS  Bob  Hope  ( Bob  Hope  class)  and  USNS  Dahl 

( Watson  class)  Large  Medium  Speed  Roll-on  roll-off 
(LMSR)  ships. 

-  Initial  observations  and  assessment  indicate  deficiencies 
exist  with  MLP  (CCS)  skin-to-skin  mooring  operations 
with  Joint  High  Speed  Vessel  (JHSV).  During  the  first 
test,  several  mooring  lines  parted,  precluding  completion 
of  the  test  event.  During  the  second  test,  the  mooring  line 
issue  was  resolved  but  the  JHSV  ramp  suffered  a  casualty, 
precluding  completion  of  the  test. 

-  The  Navy’s  Commander,  Operational  Test  and  Evaluation 
Force  (COTF)  conducted  cybersecurity  testing  of  MLP 
(CCS)  and  no  significant  vulnerabilities  were  noted. 

Initial  evaluation  suggests  that  even  if  a  cyber-adversary 
gained  access,  overall  ship’s  mission  disruption  would  be 
minimal. 

•  The  MLP  program  did  not  conduct  any  major  live  fire  test 
events  during  FY14.  The  Navy  plans  to  conduct  the  Total 
Ship  Survivability  Trial  on  the  MLP  Afloat  Forward  Staging 
Base  (AFSB)  variant  in  FY16,  which  will  provide  data  for 
recoverability  analysis.  The  Navy  issued  the  Detailed  Design 
Survivability  Assessment  Report  in  December  2013.  The  final 
Survivability  Assessment  Report  is  planned  for  FY17. 

System 

•  MLP  is  a  modified  heavy-lift  ship  the  Navy  procured  that 
uses  float-on/float-off  technology.  It  is  based  on  the  British 
Petroleum  Alaska  class  oil  tanker. 

•  Thirty-four  Military  Sealift  Command  (MSC)  contracted 
mariners  will  operate  and  maintain  the  MLP  (CCS)  vessels. 
Thirty-four  MSC  civilian  mariners  will  operate  and  maintain 
the  MLP  (AFSB)  vessels.  MSC  will  serve  as  MLP  Life  Cycle 
Managers. 

•  The  Navy  plans  to  deliver  the  MLP  with  two  different  mission 
capabilities:  a  CCS  on  hulls  1  and  2,  and  an  AFSB  on  hulls  3 
and  4. 


•  The  Navy  intends  for  CCS  to: 

-  Support  Mobile  Prepositioning  Force  operations  by 
facilitating  at-sea  transfer  and  delivery  of  pre -positioned 
assets  to  units  ashore 

-  Consist  of  a  vehicle  staging  area  (raised  vehicle  deck), 
vehicle  transfer  ramp,  large  mooring  fenders,  an 
emergency-only  commercial  helicopter  operating  spot, 
and  three  LCAC  lanes/operating  spots  (barriers,  catwalk, 
lighting,  wash  down,  and  fueling  services) 

-  Be  equipped  with  a  crane  for  placing  of  fenders  used  for 
skin-to-skin  operations  with  an  LMSR  or  JHSV 

-  Interface  with  the  LMSR  ships  and  LCACs  or  the 
follow-on  Ship-to-Shore  Connectors  to  permit 
off-loading  military  vehicles  ranging  from  High  Mobility 
Multi-purpose  Wheeled  Vehicles  to  battle  tanks  (M1A2) 

-  Be  classified  as  a  non-combatant  with  a  limited 
self-protection  capability 
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•  The  Navy  intends  for  AFSB  to: 

-  Support  Airborne  Mine  Countermeasure  (AMCM) 
operations 

-  Host  a  squadron  of  four  MH-53E  helicopters 

-  Include  a  two-spot  flight  deck,  hangar  facility, 
helicopter-fueling  capability,  ordnance  storage, 
operation  planning  and  work  spaces,  and  berthing  for 
250  Navy  personnel 

-  Have  a  mission  deck  below  the  flight  deck  with  a  crane 
for  storing  and  deploying  the  various  mine-hunting  and 
clearing  equipment  used  with  the  MH-53E  helicopters; 
explosive  ordnance  demolition  boats  and  equipment  may 
also  be  stored  and  handled  on  the  mission  deck 

-  Support  Special  Forces  in  the  future  when  funding  is 
available  to  provide  that  capability 

-  Be  classified  as  a  non-combatant  with  a  limited 
self-protection  capability 

Mission 

•  The  Navy  developed  MLP  to  host  multiple  mission  sets, 
operate  from  international  waters,  and  persist  for  extended 
periods  providing  a  capability  unencumbered  by  geo-political 
constraints  in  order  to  meet  strategic  goals. 


•  Combatant  Commanders  will  use  MLP  (CCS)  as  a  surface 
interface  between  other  Mobile  Prepositioning  Force  (future) 
squadron  ships  and  connectors  and  the  sea  base.  They  will 
also  use  it  to  transfer  personnel  and  equipment  from  LMSR 
and  JHSVs  to  the  MLP  (CCS),  and  then  onto  LCACs  or 
Ship-to-Shore  Connectors  to  facilitate  delivery  ashore  of 
forces  from  the  sea. 

•  Combatant  Commanders  will  use  MLP  (AFSB)  to  provide 
AMCM  capabilities  during  the  AMCM  mission  to  support 
the  legacy  MH-53E  helicopters  and  host  the  various 
mine-detecting  and  clearing  equipment  used  with  the 
helicopters,  along  with  explosive  ordnance  demolition  boats 
and  equipment. 

Major  Contractors 

•  MLP  base  ship  and  MLP  AFSB:  General  Dynamic’s 
National  Steel  and  Shipbuilding  Company 

(NAS  SCO)  -  San  Diego,  California 

•  CCS  arrangement :  Vigor  Marine  (Limited  Liability  Company) 
Shipbuilding  -  Portland,  Oregon 


Activity 

•  On  June  13,  2014,  DOT&E  approved  the  MLP  (CCS)  IOT&E 
test  plan.  The  test  plan  adopts  an  integrated  test  approach 
where  the  Navy  conducts  developmental  and  operational 
testing  concurrently,  with  each  having  its  own  set  of  metrics 
and  data  collection.  All  tests  were  conducted  in  accordance 
with  the  DOT&E-approved  test  plan. 

•  USNS  Montford  Point  (MLP-1)  launched  November  2012, 
completed  builders  and  acceptance  trials,  delivered 

in  May  2013,  and  completed  final  contract  trials  in 
September  2013.  The  Navy  and  Vigor  Marine  completed 
installation  of  the  CCS  in  April  2014,  and  PDT&T 
commenced  in  June  2014.  USNS  Lewis  B.  Puller  (MLP-3 
AFSB)  is  expected  to  deliver  in  4QFY15.  The  following  test 
events  were  conducted  at  sea  off  Camp  Pendleton,  in  Long 
Beach  Harbor,  and  in  harbor  in  San  Diego,  California: 

-  The  Navy  and  Marine  Corps  attempted  to  conduct  a 
day  and  night  interface  test  mooring  of  the  JHSV  to  the 
MLP  (CCS)  vessel  during  the  week  of  June  23,  2014. 
USNS  Millinocket  (JHSV-3)  moored  skin-to-skin  with 
USNS  Montford  Point  (MLP-1  CCS)  and  Marine  Corps 
vehicles  transited  back  and  forth  during  daylight.  The 
vehicle  transfer  in  daylight  was  completed  successfully, 
but  five  mooring  lines  broke  prompting  cancellation  of  the 
night  test. 

-  The  Navy  and  Marine  Corps  attempted  the  same  evolution 
a  second  time  on  October  29,  2014.  Although  the  mooring 
line  problem  was  resolved,  and  both  day  and  night 
moorings  were  completed,  the  JHSV  ramp  suffered  a 


casualty,  prompting  cancellation  of  further  vehicle  transfers 
after  recovering  the  first  vehicle. 

-  The  Navy  and  Marine  Corps  conducted  a  portion  of  the 
primary  timed  vehicle  transfer  requirement  during  a  fleet 
exercise  the  week  of  July  7,  2014.  The  primary  timed 
requirement  for  MLP  (CCS)  is  to  enable  the  transfer 
ashore  of  all  the  vehicles  for  a  Marine  Corps  reinforced 
rifle  company  in  a  12-hour  period  from  25  nautical  miles 
at  sea  with  the  exception  of  Amphibious  Assault  Vehicles 
(AAVs).  AAVs  are  slower  than  LCACs  in  the  water,  and 
are  best  launched  within  five  nautical  miles  from  the  beach. 
During  this  exercise,  MLP-1  was  positioned  just  inside 

3  nautical  miles  from  shore,  and  the  United  States  Marine 
Corps  transferred  15  AAVs  ashore  in  1  hour  and  4  minutes. 

-  The  Navy  and  Marine  Corps  successfully  completed  the 
rest  of  the  reinforced  rifle  company  vehicle  transfers  via 
LCAC  in  a  timed  event  on  October  14,  2014.  Although 
MLP-1  was  not  positioned  25  nautical  miles  from  shore, 
the  LCACs  transited  a  route  of  25  nautical  miles  both  to 
shore  and  then  back  to  MLP- 1  for  the  next  load. 

-  The  Navy  conducted  a  24-hour,  15-knot  fuel  economy  trial 
as  MLP-1  transited  from  the  Seattle  Washington  area  to 
Southern  California  waters.  Preliminary  analysis  indicates 
the  ship  can  easily  achieve  the  9,500  nautical  miles, 
unrefueled,  15-knot  transit  requirement. 

-  The  Navy  and  Marine  Corps  successfully  conducted 
skin-to-skin  operations  through  Sea  State  3  with  both  the 
USNS  Bob  Hope  ( Bob  Hope  class)  and  the  USNS  Dahl 
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( Watson  class)  LMSRs,  to  include  vehicle  transfer  and 
LCAC  operations,  in  various  tests  during  September  and 
October  2014. 

-  The  Navy  and  Marine  Corps  conducted  a  Critical  System 
Maintenance  Review  with  the  ship’s  company  to  assist  in 
evaluating  suitability  of  both  maintenance  and  logistics  for 
the  ship  class. 

-  The  Navy  conducted  a  limited  self-defense  drill  (no  targets 
engaged)  and  a  Structural  Test  Fire  Event  that  verified 
fields  of  fire  and  0.50  caliber  machine  gun  mount  structure 
suitability  on  November  3,  2014. 

•  COTF  conducted  a  local  cybersecurity  evaluation  of  MLP-1 
during  the  week  of  July  28,  2014.  COTF  conducted  this 
evaluation  to  assess  the  ship’s  hosts  and  servers  on  the 
unclassified  and  classified  networks. 

•  COTF  conducted  the  remote  reconnaissance  and  cyber-attack 
evaluation  on  MLP-1  on  August  25  -  26,  2014.  This 
evaluation  focused  on  how  initial  footholds  can  be  gained  by 
an  external  attacker  and  how  well  they  can  be  leveraged. 

•  The  MLP  program  did  not  conduct  any  major  live  fire  test 
events  during  FY14.  The  Navy  plans  to  conduct  the  Total 
Ship  Survivability  Trial  on  the  MLP  AFSB  variant  in  FY16, 
which  will  provide  data  for  recoverability  analysis.  The  Navy 
issued  the  Detailed  Design  Survivability  Assessment  Report  in 
December  2013.  The  final  Survivability  Assessment  Report  is 
planned  for  FY17. 

Assessment 

•  This  report  provides  only  a  preliminary  assessment  of  the 
MLP  (CCS)  based  on  observations  on  USNS  Montford  Point 
(MLP-1).  DOT&E  will  provide  the  final  assessment  in  the 
3QFY15  IOT&E  report. 

•  Vehicle  transfer  at  sea  with  JF1SV  moored  skin-to-skin  with 
MLP  (CCS)  is  not  advised  (mooring  with  JF1SV  is  a  secondary 
mission  for  MLP.)  Initial  observations  and  assessment  indicate 
deficiencies  exist  with  MLP  (CCS)  skin-to-skin  mooring 


operations  with  JFISV.  During  the  first  test  event,  several 
mooring  lines  parted,  precluding  completion  of  the  test  event. 
During  the  second  test  event,  the  mooring  line  issue  was 
resolved  but  the  JFISV  ramp  suffered  a  casualty,  precluding 
completion  of  the  test.  Analysis  is  in  progress  and  final 
assessment  will  be  provided  in  the  IOT&E  report. 

•  The  primary  timed  vehicle  transfer  requirement  is  satisfied. 
The  unrefueled  range  requirement  is  satisfied 

•  The  Navy  demonstrated  skin-to-skin  operations  and  vehicle 
transfer  through  Sea  State  3  with  both  the  USNS  Bob  Hope 
(Bob  Hope  class)  and  USNS  Dahl  ( Watson  class)  LMSRs. 

•  The  local  cybersecurity  test  demonstrated  that  the  network’s 
Host-Based  Security  System  stopped  most  of  COTF’s 
cyber-attacks  against  unclassified  and  classified  networks.  As 
the  ship’s  networks  are  not  connected  to  the  ship’s  critical 
systems,  the  loss  of  either  unclassified  or  classified  networks 
during  operations  would  be  an  inconvenience,  but  would  not 
hinder  the  ship’s  ability  to  conduct  its  mission  since  it  has 
communication  backups,  including  radios  and  standalone 
satellite  phones. 

•  During  the  remote  reconnaissance  and  cyber-attack  evaluation, 
COTF  was  unable  to  gain  a  foothold  on  the  MLP-1  networks 
with  the  toolset  used  for  these  assessments.  However,  the  test 
did  not  explore  the  vulnerability  of  the  ship  to  very  advanced 
cyber  threats  due  to  security  restrictions  in  place  during  the 
time  of  the  test. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendation.  IOT&E  analysis  is  ongoing 
and  DOT&E  expects  to  issue  a  full  assessment  and 
recommendations  in  3QFY15. 

1.  The  Navy  should  reevaluate  the  need  for  at  sea  skin-to-skin 
operations  between  MLP  (CCS)  and  JHSV. 
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MQ-4C  Triton  Unmanned  Aircraft  System 


Executive  Summary 

•  In  1QFY14,  USD(AT&L)  approved  a  revised  Triton  program 
plan,  which  projects  an  operational  assessment  (OA)  occurring 
in  3QFY15  and  a  Milestone  C  in  1QFY15.  An  updated 
Acquisition  Program  Baseline  was  subsequently  approved  on 
July  7,  2014.  The  revised  program  schedule  has  allowed  the 
program  to  execute  additional  risk  reduction  testing  prior  to 
conducting  the  OA. 

•  The  Navy  continued  several  development  efforts  in  FY14, 
including  ground  and  lab  testing  of  system  communications 
and  datalink  architectures,  and  the  Multi-Function  Active 
Sensor  (MFAS)  radar  risk  reduction  flight  testing  on  a 
Northrop  Grumman  surrogate  test  bed  aircraft. 

•  On  September  18,  2014,  Triton  aircraft  B-l  was  ferried 
from  Edwards  AFB,  California,  to  Patuxent  River  Naval  Air 
Station  (NAS),  Maryland.  Basing  of  Triton  at  Patuxent  River 
NAS  is  a  prerequisite  for  integration  of  mission  systems  on 
the  air  vehicle,  development  of  the  Integrated  Functional 
Capability  2.2  system  software,  and  operational  test  flights. 
The  movement  of  the  Triton  Aircraft  from  Edwards  AFB 

to  Patuxent  River  NAS  was  an  important  milestone  for  the 
program. 

System 

•  The  MQ-4C  Triton  Unmanned  Aircraft  System  (UAS) 
is  an  Intelligence,  Surveillance,  and  Reconnaissance 
system-of-systems  consisting  of  the  high-altitude, 
long-endurance  MQ-4C  air  vehicle,  sensor  payloads,  and 
supporting  ground  control  stations.  The  MQ-4C  system 
is  a  part  of  the  Navy  Maritime  Patrol  and  Reconnaissance 
family-of-systems,  with  capabilities  designed  to  complement 
the  P-8A  Poseidon  Multi-mission  Maritime  aircraft. 

•  The  MQ-4C  air  vehicle  design  is  based  on  the  Air 
Force  RQ-4B  Global  Hawk  air  vehicle  with  significant 
modifications  that  include  strengthened  wing  structures  and 
an  anti-ice  and  de-icing  system.  An  air  traffic  de-confliction 
and  collision  avoidance  radar  system  is  also  planned,  but  the 
Navy  is  no  longer  pursuing  the  Air-to-Air  Radar  Subsystem 
(AARSS). 

•  The  MQ-4C  is  equipped  with  the  MFAS  maritime  surveillance 
radar  to  detect,  identify,  and  track  surface  targets  and  produce 
high-resolution  imagery.  The  MQ-4C  electro  optical/infrared 


sensor  provides  full  motion  video  and  still  imagery  of  surface 
targets.  An  Electronic  Support  Measures  system  detects, 
identifies,  and  geolocates  radar  threat  signals.  An  Automatic 
Identification  System  receiver  permits  the  detection, 
identification,  geolocation,  and  tracking  of  cooperative  vessels 
equipped  with  Automatic  Identification  System  transponders. 

•  Onboard  line-of-sight  and  beyond  line-of-sight  datalink 
systems  transmit  sensor  data  from  the  air  vehicle  to  ground 
control  stations  for  dissemination  to  fleet  tactical  operation 
centers  and  intelligence  exploitation  sites. 

Mission 

•  Commanders  use  units  equipped  with  MQ-4C  to  conduct 
maritime  surveillance  operations  and  provide  high-  and 
medium-altitude,  long-endurance  intelligence  collection. 

•  MQ-4C  operators  detect,  identify,  track,  and  assess  maritime 
and  littoral  targets  of  interest  and  collect  imagery  and  signals 
intelligence  information.  Operators  disseminate  sensor  data 
to  fleet  units  to  support  a  wide  range  of  maritime  missions 

to  include  surface  warfare,  intelligence  operations,  strike 
warfare,  maritime  interdiction,  amphibious  warfare,  homeland 
defense,  and  search  and  rescue. 

Major  Contractor 

Northrop  Grumman  Aerospace  Systems,  Battle  Management  and 

Engagement  Systems  Division  -  Rancho  Bernardo,  California 


Activity 

•  Technical  problems  and  budget  impacts  caused  delays  in  the 
flight  test  schedule  throughout  FY12  and  FY13.  In  1QFY14, 
the  USD(AT&L)  approved  the  Triton  program  to  re -baseline 


the  schedule  and  budget.  The  current  schedule  reflects  a  goal 
to  conduct  an  OA  in  3QFY15  and  to  reach  Milestone  C  in 
1QFY16. 
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•  Since  May  2013,  the  Navy  has  completed  a  total  of  16  flights, 
including  the  ferry  flights  from  Edwards  AFB,  with  9  of  those 
flights  completed  in  FY14. 

•  During  the  developmental  testing  and  envelope  expansion 
of  the  Air  Vehicle,  the  Navy  continued  several  development 
efforts  including: 

-  Ground  and  lab  testing  of  system  communications  and 
datalink  architectures,  which  are  intended  to  provide  risk 
reduction  for  mission  systems  and  end-to-end  mission 
flight  testing. 

-  MFAS  radar  risk  reduction  flight  testing  on  a  Northrop 
Grumman  surrogate  test  bed  aircraft  to  identify  and  resolve 
potential  radar  performance  problems  prior  to  integration 
on  the  MQ-4C  air  vehicle.  The  contractor  has  completed  a 
total  of  39  total  flights  (8  in  CY14)  and  implemented  radar 
software  changes  to  improve  sensor  stability,  maritime 
target  surveillance,  tracking  performance,  and  synthetic 
aperture  radar  image  quality. 

•  Due  to  technical  difficulties  with  the  AARSS,  the  Navy  is 
no  longer  pursuing  AARSS  as  a  technical  solution  to  the  due 
regard  or  “sense  and  avoid”  capability  for  Triton.  The  Navy 
continues  to  pursue  procedural  and/or  technical  solutions  to 
compensate  for  the  loss  of  the  AARSS. 

•  The  Navy  has  developed  a  network  environment  that  can  be 
used  for  operational  training  and  employment.  In  addition,  the 
Navy  has  completed  construction  of  the  hangar  and  training 
spaces  required  to  execute  the  Triton  test  program. 

•  On  September  18,  2014,  Triton  B-l  aircraft  was  ferried  from 
Edwards  AFB,  California,  to  Patuxent  River  NAS,  Maryland. 

Assessment 

•  The  re -baseline  of  the  program  has  provided  stability  to  the 
Triton  program  and  enabled  the  Navy  to  execute  acquisition 
and  evaluation  strategies  in  FY14,  and  to  execute  additional 
risk  reduction  testing  prior  to  conducting  the  OA. 

•  Developmental  testing  of  the  Air  Vehicle  revealed  the 
following: 

-  Ground  and  lab  testing  of  system  communications  and 
datalink  architectures  are  producing  early  deficiency 
reports  in  support  of  efforts  to  improve  system  maturity 
before  mission  system  flight  testing  begins. 


-  The  Northrop  Grumman  MFAS  risk  reduction  flight  test 
program  identified  several  system  performance  problems 
for  resolution  prior  to  MFAS  integration  on  to  the  MQ-4C 
platform.  The  program  has  implemented  radar  software 
changes  to  improve  sensor  stability,  maritime  target 
surveillance  and  tracking  performance,  and  synthetic 
aperture  radar  image  quality.  Other  UAS  platforms  have 
experienced  degradation  in  performance  when  sensors 
move  from  surrogate  platforms  to  the  developmental 
aircraft.  It  is  likely  that  some  degradation  is  possible  with 
Triton  as  well,  but  the  continuing  MFAS  test  flights  on  the 
surrogate  have  reduced  the  risk  of  initial  integration. 

•  Traffic  de-confliction  and  collision  avoidance  provide  critical 
mission  capability  for  operation  of  the  MQ-4C  in  civil  and 
international  airspace  in  support  of  global  naval  operations 

•  Prior  to  beginning  developmental  testing  of  mission  systems, 
the  Navy  must  also  integrate  sensors  onto  the  air  vehicle  at 
Patuxent  River  NAS,  complete  development  of  the  Integrated 
Functional  Capability  2.2  system  software,  and  load  it  on  the 
Triton  system. 

•  The  movement  of  the  Triton  air  vehicle  from  Edwards  AFB 
to  Patuxent  River  NAS  was  an  important  milestone  for  the 
program. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  program  was 
re -baselined  in  December  2013,  which  affected  program 
scheduling  and  delayed  Milestone  C  and  completion  of 
the  Test  and  Evaluation  Master  Plan  (TEMP)  update,  as 
recommended  in  FY13.  The  Navy  still  needs  to  develop  a 
revised  test  schedule  and  TEMP  that  reflect  test  delays. 

•  FY 14  Recommendations.  The  Navy  should: 

1.  Prepare  and  submit  a  TEMP  update  to  DOT&E  by  the 
Milestone  C  decision. 

2.  The  Navy  should  complete  the  planned  OA  in  FY15  in 
preparation  for  Milestone  C. 

3.  During  the  planned  FY15  OA,  the  Navy  should  demonstrate 
tactics  and  procedures  that  will  enable  Triton  to  descend 
and  operate  in  medium-  and  low-altitude  environments,  in 
accordance  with  the  MQ-4C  Triton  Warfighting  Concept  of 
Operations. 
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Multi-Static  Active  Coherent  (MAC)  System 


Executive  Summary 

•  The  Navy  completed  the  initial  operational  testing  of  the 
Multi-Static  Active  Coherent  (MAC)  system  on  P-3C  aircraft 
in  October  2013. 

•  Operational  test  results  indicate  that  the  MAC  system 
provides  P-3C  aircraft  with  a  wide-area  Anti-Submarine 
Warfare  (ASW)  search  capability  in  select  scenarios  in  some 
environments  but  it  does  not  meet  the  program’s  requirements 
in  other  operational  environments  or  scenarios. 

•  The  IOT&E  did  not  fully  examine  the  capability  of  MAC 
across  all  operational  conditions,  representative  operational 
environments,  and  target  types.  DOT&E  agreed  to  limit 
testing  during  the  initial  phase  because  sufficient  active  source 
buoys  were  not  available  and  because  the  MAC  system  would 
be  installed  and  further  tested  on  P-8A  aircraft  in  several 
increments  through  FY19. 

•  In  FY14,  the  Navy  installed  the  MAC  system  on  the  P-8A 
Poseidon  Multi-mission  Maritime  aircraft.  Due  to  integration 
problems,  the  Navy  delayed  the  initial  MAC  operational 
testing  on  the  P-8A  from  March  2014  to  November  2014. 

System 

•  The  MAC  system  is  an  active  sonar  system  composed  of 
two  types  of  sonobuoys  (source  and  receiver)  and  an  acoustic 
processing  and  aircraft  mission  computer  software  suite.  It  is 
employed  by  the  Navy’s  maritime  patrol  aircraft  (P-3Cs  and 
eventually  P-8As)  to  search  for  and  locate  threat  submarines 
in  a  variety  of  ocean  conditions.  To  plan  MAC  missions,  the 
Navy  has  updated  the  Active  System  Performance  Estimate 
Computer  Tool  (ASPECT)/Multi-static  Planning  Acoustics 
Toolkit  (MPACT)  previously  used  to  plan  Improved  Extended 
Echo  Ranging  (IEER)  system  missions. 

•  MAC  replaces  the  Navy’s  current  IEER  system,  which 
employs  non-coherent  sources  to  produce  loud  sounds 
that  reflect  off  submarine  targets.  MAC  employs  new 
coherent  source  buoys  that  enable  multiple  pings,  optimized 
waveforms,  and  various  ping  durations,  none  of  which  the 
legacy  IEER  system  provided. 


•  The  Navy  initially  intends  to  employ  MAC  on  P-3C  aircraft 
in  a  limited  set  of  acoustic  environments.  Future  increments 
of  MAC  will  be  employed  on  P-8A  aircraft  and  in  a  wider 
variety  of  acoustic  ocean  environments  in  order  to  span  the 
operational  envelope  of  threat  submarine  operations.  MAC 
will  be  the  primary  wide-area  acoustic  search  system  for  the 
P-8A. 

•  MAC  is  expected  to  have  fewer  effects  on  marine  mammals 
and  the  environment  than  the  legacy  IEER  system. 

Mission 

The  Navy  intends  for  P-3C  and  P-8A  crews  equipped  with  MAC 

to  support  the  search,  detect,  and  localization  phases  of  the 

ASW  mission.  MAC  is  particularly  focused  on  large-area  active 

acoustic  searches  for  threat  submarines. 

Major  Contractors 

•  Lockheed  Martin  -  Manassas,  Virginia 

•  Sparton  Electronics  Florida,  Inc.  -  De  Leon  Springs,  Florida 

•  Ultra  Electronics,  Undersea  Sensor  Systems  Incorporated 
(USSI)  -  Columbia  City,  Indiana 

•  Boeing  Defense,  Space,  and  Security  -  St.  Louis,  Missouri 


Activity 

•  The  Navy  completed  operational  testing  of  the  MAC  Phase  1 
system  on  P-3C  Multi-mission  Aircraft  in  October  2013. 
Operational  testing  consisted  of  3  developmental  test  events 
conducted  off  the  coast  of  Jacksonville,  Florida;  7  dedicated 
operational  test  events  conducted  in  the  Southern  California 
Fleet  Operating  Areas  (SOCAL);  and  14  events  in  the 
Narragansett  Bay  Operating  Areas  (NBOA).  Testing  did  not 
include  the  ASPECT/MPACT  because  its  bottom  enviromnent 


database  was  poorly  populated  causing  it  to  inaccurately 
predict  the  probability  of  detection.  The  Navy  conducted  the 
operational  testing  in  accordance  with  a  DOT&E-approved 
test  plan. 

•  Following  the  first  five  NBOA  test  events,  the  Navy  paused 
operational  testing  to  investigate  observed  performance 
problems.  The  Navy  identified  operator  training  and  material 
problems  on  the  P-3C  aircraft  as  probable  causes  of  the 
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degraded  performance.  The  Navy  required  these  events  to  be 
repeated. 

•  DOT&E  issued  an  IOT&E  report  in  July  2014. 

•  In  FY14,  the  Navy  installed  the  MAC  system  on  the  P-8A 
Poseidon  aircraft.  Due  to  integration  problems,  the  Navy 
delayed  the  initial  MAC  operational  testing  on  the  P-8A  from 
March  2014  to  November  2014. 

•  The  Navy  and  DOT&E  commenced  developing  a  Test  and 
Evaluation  Master  Plan  to  identify  deferred  MAC  testing  and 
to  plan  for  the  Phase  2  MAC  improvements.  To  efficiently 
utilize  test  resources,  DOT&E  is  requiring  the  test  program 
be  consistent  with  and  utilize  MAC  events  programmed  in 
the  recently  approved  P-8A  Increment  2  Test  and  Evaluation 
Master  Plan. 

•  The  Navy  started  operational  testing  of  the  MAC  installed  on 
the  P-8  A  aircraft  off  the  coast  of  Florida  in  November  2014. 

Assessment 

•  Operational  test  results  indicate  that  the  MAC  system  provides 
P-3C  aircraft  with  a  wide-area  ASW  search  capability  in  some 
environments  and  for  select  target  scenarios,  but  MAC  falls 
short  of  what  the  fleet  identified  as  the  capability  needed  to 
protect  high-value  units.  Initial  testing  identified  that  detection 
performance  strongly  relies  on  the  characteristics  of  the  ocean 
environment  and  the  tactics  employed  by  the  target  to  evade 
detection.  Testing  to  understand  the  effects  different  threat 
types  and  environments  have  on  performance  will  continue 
through  FY19  in  conjunction  with  the  P-8  program. 

•  The  IOT&E  did  not  fully  examine  the  capability  of  MAC 
across  all  operational  conditions,  representative  operational 
environments,  and  target  types.  DOT&E  agreed  to  limit 
testing  of  the  initial  phase  of  MAC  because  sufficient  active 
source  buoys  were  not  available  and  because  the  MAC  system 
would  be  installed  and  tested  on  P-8A  aircraft  beginning  in 
FY14  and  continuing  through  FY19. 

•  Although  the  MAC  system  demonstrated  detection  capability 
against  evasive  undersea  targets,  acoustic  operators  were 
expected  to  quickly  distinguish  system  submarine  detections 
from  a  variety  of  non-submarine  clutter  detections,  some  of 
which  appeared  target-like.  Complicating  this  task,  completed 
test  analysis  identified  that  the  MAC  system  detections 

of  target  and  non-target  clutter  varies  with  environmental 
conditions  and  likely  target  types.  The  data  also  show 
operators  are  only  able  to  recognize  a  small  fraction  of  valid 
system  submarine  detections  as  a  possible  target  and  spent 
time  assessing  and  prosecuting  false  targets. 

•  The  Navy  uses  ASPECT/MPACT  to  develop  MAC  search 
plans  and  to  estimate  theoretical  system  performance.  In 
addition  to  the  known  shortfalls  with  the  environmental 
databases  used  by  ASPECT  that  the  Navy  deferred,  the 
planning  tool  performance  estimates  are  highly  dependent  on 
the  wide-range  of  potential  mission  planning  input  parameters 
estimated  by  the  mission  planner.  As  a  result,  ASPECT 
performance  estimates  can  widely  vary  when  compared  to  test 


results.  Since  ASPECT  does  not  have  a  good  estimate  for  the 
operator  recognition  of  the  submarine  target,  it  overestimates 
ASW  detection  performance. 

•  For  additional  information,  see  DOT&E’s  classified  IOT&E 
report  on  the  MAC  System  on  P-3C  Aircraft  dated  July  2014. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  partially 
addressed  the  FY13  recommendations. 

1.  The  Navy  Program  Office  is  investigating  fleet  exercise 
data  to  assess  detection  performance  and  to  gather  data  for 
developing  future  algorithm  and  software  improvements. 
Although  fleet  exercise  data  includes  new  environments 
where  the  fleet  operates  in  peacetime,  the  Navy  has  not 
investigated  MAC  performance  variability  with  a  variety  of 
submarine  target  types. 

2.  The  Navy  has  not  completed  development  of  a  sustainable 
MAC  training  program  or  completed  the  formal  updates  to 
tactics  guidelines  and  documentation. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Plan  and  complete  the  outstanding  MAC  operational 
testing  and  investigate  MAC  system  and  operator 
performance  against  different  target  types.  This  testing 
should  be  in  conjunction  with  P-8AMAC  introduction  and 
improvements. 

2.  Implement  the  recommendations  in  DOT&E’s  IOT&E 
report.  DOT&E  provided  15  recommendations  to  improve 
the  MAC  system  performance  and  5  recommendations  to 
improve  test  realism,  minimize  test  limitations,  and  improve 
data  collection.  Significant  unclassified  recommendations 
include — 

-  Investigate  and  develop  improved  methods  for  aircrews 
to  sample  and  characterize  the  time  and  space  variability 
of  the  search  area  environment. 

-  Investigate  and  develop  tactics  to  improve  the  operator’s 
ability  to  transition  system  detections  to  high  confidence 
target  detection.  Consider  measures  to  balance  operator 
workload  and  update  search  plans  based  on  the  actual 
conditions  experienced  in  the  search  area. 

-  Investigate  the  system’s  capability  for  longer  range 
detections  based  on  the  environmental  conditions  in  the 
search  area. 

-  Investigate  and  develop  improvements  to  the  ASPECT 
planning  system  and  the  supporting  databases. 

-  Complete  the  MAC  upgrades  to  aircrew  trainers  and 
training  documentation. 

-  Improve  the  operator’s  capability  to  utilize  the  passive 
detection  capability  of  the  MAC  receiver  buoys. 

-  Conduct  future  testing  and  exercises  using  a  variety  of 
target  surrogates  that  execute  tactics  appropriate  for 
their  assigned  mission.  The  targets  and  tactics  should  be 
validated  as  representative  of  the  threats. 


228  MAC 


F Y1  4  NAVY  PROGRAMS 


MV-22  Osprey 


Executive  Summary 

•  The  Navy  conducted  cyber  testing  of  the  MV-22B  Joint 
Mission  Planning  System  -  Maritime  (JMPS-M)  in 
August  2013  and  FOT&E  OT-IIIJ  from  September  through 
October  2013. 

•  The  Navy  should  implement  stronger  role-based  access 
controls  for  JMPS-M  to  strengthen  cybersecurity. 

•  OT-IIIJ  demonstrated  the  utility  of  most  system  hardware  and 
flight  software  improvements. 

•  Additional  testing  is  needed  to  demonstrate  the  utility 
of  the  APX-123  Mode  5  Identification  Friend  or  Foe 
(IFF)  Transponder  and  the  Integrated  Waveform  Satellite 
Communications  performance,  and  the  reliability  of  the 
GAU-21  Defensive  Weapon  System. 

•  The  Navy  should  continue  to  execute  the  reliability  growth 
program  for  the  MV-22  fleet 

System 

•  There  are  two  variants  of  the  V-22:  the  Marine  Corps  MV-22 
and  the  Air  Force/U.S.  Special  Operations  Command  CV-22. 
The  air  vehicles  for  Air  Force  and  Marine  Corps  missions 
are  nearly  identical,  with  common  subsystems  and  military 
components  sustainable  by  each  Service’s  logistics  system. 

•  The  Marine  Corps  is  replacing  the  aging  CH-46  and  CH-53D 
helicopters  with  MV-22s.  The  MV-22  is  a  tilt  rotor  aircraft 
capable  of  conventional  wing-borne  flight  and  vertical  take-off 
and  landing. 

•  The  MV-22  can  carry  24  combat-equipped  Marines  and 
operate  from  ship  or  shore.  It  can  carry  an  external  load  up 
to  10,000  pounds  over  50  nautical  miles,  and  can  self-deploy 
2,363  nautical  miles  with  a  single  aerial  refueling. 

•  Recent  system  upgrades  include  the  following: 

-  Enhanced  Rapid  Ground  Refueling  system 

-  APX-123  IFF  transponder  (replaced  the  APX-118  IFF 
transponder  to  support  Mode  5) 

-  Generation  5  radios  (replaced  Generation  3  radios  to 
support  Integrated  Waveform  Satellite  Communications 
(SATCOM)) 

-  GAU-17  Defensive  Weapon  System  improvements 
(upgraded  with  a  sensor-only  mode  that  allows  the  gunner 
to  use  the  electro-optical  sensor  when  the  gun  turret  is  not 
being  used) 


-  Modified  aircraft  flight  control  laws  (granted  pilots  greater 
lateral  control  authority  and  increased  thrust  sensitivity) 

-  Increased  Forward-Looking  Infrared  Sensor  look  down 
angle  (expanded  by  a  factor  of  six) 

-  Blue  Force  Tracker  (BFT-1)  mount  and  circuitry 
improvements 

Mission 

•  Squadrons  equipped  with  MV-22s  will  provide  medium-lift 
assault  support  in  the  following  operations: 

-  Ship-to-Objective  Maneuver 

-  Sustained  operations  ashore 

-  Tactical  recovery  of  aircraft  and  personnel 

-  Self-deployment 

-  Amphibious  evacuation 

•  Currently  deployed  squadrons  are  providing  high-tempo 
battlefield  transportation  in  the  U.S.  Central  Command  Area  of 
Responsibility. 

Major  Contractors 

Bell-Boeing  Joint  Venture: 

•  Bell  Helicopter  -  Amarillo,  Texas 

•  The  Boeing  Company  -  Ridley  Township,  Pennsylvania 


Activity 

•  The  Navy,  with  assistance  from  the  Marine  Corp  Information 
Assurance  Red  Team,  conducted  cyber  testing  of  the 
MV-22B  JMPS-M  in  August  2013.  The  JMPS-M  provides  a 
network-based  mission-planning  environment  for  the  MV-22 
aircraft,  and  is  a  potential  cyber  vulnerability  for  the  MV-22B. 


•  The  Marine  Corps  Operational  Test  and  Evaluation 
Squadron  22  (VMX-22)  conducted  FOT&E  OT-IIIJ  from 
September  through  October  2013.  This  89-flight  hour 
dedicated  operational  test  was  preceded  by  14  months  and 
627  flight  hours  of  integrated  testing  (1T-1IID). 
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-  Operational  pilots  evaluated  capabilities  of  the  latest 
MV-22B  enhancements:  the  enhanced  Rapid  Ground 
Refueling  system,  APX-123  IFF  transponder,  Generation  5 
radios,  GAU-17  Defensive  Weapon  System  improvements, 
increased  Forward-Looking  Infrared  Sensor  look  down 
angle,  BFT-1  mount  and  circuitry  improvements,  and 
modified  aircraft  flight  control  software. 

-  VMX-22  conducted  the  OT-IIIJ  missions  using  four 
production-representative  aircraft  (two  Block  C  and 
two  Block  B)  at  Marine  Corps  Air  Station  New  River, 
North  Carolina,  and  Petersen  AFB,  Colorado.  DOT&E 
observed  all  of  the  OT-I11J  missions. 

-  OT-IIIJ  was  conducted  in  accordance  with  a 
DOT&E-approved  Test  and  Evaluation  Master  Plan  and 
Operational  Test  Plan. 

Assessment 

•  Units  equipped  with  the  MV-22B  remain  operationally 
effective,  suitable,  and  survivable  as  previously  reported. 

•  OT-IIIJ  demonstrated  the  utility  of  most  software  version 
B5.01/C2.01  and  hardware  enhancements. 

-  As  a  mobile-refueling  platform,  the  MV-22  can  now 
deliver  twice  as  much  fuel  at  twice  the  previous  refueling 
rate  during  ground  refueling  operations. 

-  The  software  version  B5.01/C2.01  improvements  aided 
mission  management  as  intended,  but  created  minor 
increases  in  pilot  workload  with  nuisance  warnings  and 
uncommanded  resets. 

•  Improvements  to  the  GAU-17  Defensive  Weapon  System,  the 
APX-123  Mode  5  IFF,  and  the  Integrated  Waveform  SATCOM 
were  not  adequately  demonstrated  during  OT-IIIJ;  following 
successful  developmental  testing,  these  will  be  tested  in 
OT-IIIK  in  2015. 

-  Only  1,500  rounds  were  fired  with  the  GAU-17  Defensive 
Weapon  System;  too  few  rounds  to  observe  the  effects  of 
the  improvements. 

-  Mode  5  IFF  interrogators  were  not  employed  during 
operational  testing. 

-  The  new  capability  of  the  Generation  5  radio  was  the 
Integrated  Waveform  SATCOM,  but  this  waveform  did  not 
work  at  all  in  developmental  or  operational  testing  of  the 
radio  on  MV-22  aircraft. 

•  Reliability  improvements  were  evident  in  the  OT-IIIJ  test 
aircraft  and  MV-22B  fleet,  but  recurring  problems  continue  to 
degrade  non-deployed  fleet  and  test  aircraft  availability. 


-  Overheating  prop-rotor  gearboxes  and  flight  control 
actuator  failures  were  noteworthy  sources  of  OT-IIIJ 
aircraft  downtime  and  maintenance  effort. 

-  Reliability  improvements  to  the  Icing  Protection  System 
(IPS)  have  been  demonstrated  on  CV-22  aircraft,  but  the 
full  set  of  IPS  upgrades  has  not  been  implemented  on  the 
MV-22B  test  aircraft.  The  reliability  of  MV-22B  IPS  could 
not  be  accurately  measured  because  of  unresolved  IPS 
reliability  failures  on  the  OT-IIIJ  test  aircraft. 

•  Information  generated  by  the  JMPS-M  is  vulnerable  to 
alteration  by  malicious  or  unwitting  users  who  gain  access  to 
administrative  functions.  Otherwise,  JMPS-M  information 
protections  successfully  prevented  unauthorized  access  by 
internal  and  external  cyber  threats. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  made 
progress  on  the  F Y 1 1  recommendation  to  improve  reliability 
of  the  IPS.  While  the  Navy  has  not  implemented  the  full  set 
of  IPS  upgrades  on  all  the  MV-22  aircraft,  it  has  demonstrated 
reliability  improvements  to  the  IPS  on  CV-22  aircraft,  which 
is  identical  to  the  system  used  on  MV-22.  The  Navy  should 
continue  to  make  reliability  improvements  through  execution 
of  its  reliability  growth  program.  The  Navy  did  not  field 

the  Traffic  and  Collision  Avoidance  System  as  DOT&E 
recommended  in  FY12. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Maintain  the  V-22  program’s  focus  on  reliability  growth, 
parts  provisioning,  and  reduction  of  repair  time  for 
gearboxes  and  flight  control  components. 

2.  Implement  role-based  access  controls  for  authorized 
JMPS-M  users  and  investigate  the  operational  effects  of 
cyber  penetration  of  this  and  other  interfacing  systems  on 
the  MV-22B. 

3.  Conduct  additional  operational  testing  of  the  GAU-17 
Defensive  Weapon  System  and  Integrated  Waveform 
SATCOM. 

4.  Include  MV-22  in  the  next  Joint  Operational  Test  Approach 
testing  of  Mode  5  IFF,  currently  scheduled  for  FY16. 

5.  Provide  appropriate  warnings  when  fielding  MV-22B 
software  versions  B5.01  and  C2.01.  In  future  software 
development,  address  nuisance  warnings  and  uncommanded 
resets. 
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P-8A  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA) 


Executive  Summary 

•  The  Navy  conducted  the  P-8A  Increment  1  FOT&E 
from  October  2013  through  March  2014  to  evaluate  the 
operational  effectiveness  and  suitability  of  the  integration  and 
employment  of  the  AGM-84D  Block  1C  Harpoon  anti-ship 
missile  system.  FOT&E  also  included  evaluation  of  software 
improvements  intended  to  correct  1 7  operationally  significant 
system  deficiencies  that  degraded  Anti-Surface  Warfare 
(ASuW);  Intelligence,  Surveillance,  and  Reconnaissance 
(ISR);  and  Command,  Control,  and  Communication  (C3) 
mission  capabilities  during  IOT&E.  Based  on  FOT&E  results, 
DOT&E  concluded  that: 

-  The  P-8A  Increment  1  system  provides  an  operationally 
effective  armed  ASuW  mission  capability  to  detect, 
classify,  and  track  maritime  surface  targets  and  engage 
them  using  the  AGM-84D  Block  1C  Harpoon  anti-ship 
missile.  System  software  changes  improved  overall 
surface  target  search,  classification,  and  tracking 
capabilities  as  compared  to  observed  IOT&E  performance. 

-  Based  on  FOT&E  results,  the  P-8  A  Increment  1  system 
does  not  yet  provide  an  operationally  effective  ISR  mission 
capability.  System  software  improvements  corrected 
several  high-priority  sensor  integration  problems  and 
improved  imagery  dissemination  capabilities.  However, 
persistent  synthetic  aperture  radar  (SAR)  image  quality 
problems,  unresolved  electronic  intelligence  collection 
deficiencies,  and  data  dissemination  limitations  continue  to 
degrade  P-8AISR  mission  capabilities.  Additional  SAR, 
electronic  intelligence  collection,  and  data  dissemination 
upgrades  are  scheduled  for  inclusion  in  upcoming 
pre-planned  software  updates  and  will  be  assessed  during 
future  FOT&E  periods. 

-  The  P-8A  Increment  1  system  provides  a  C3  mission 
capability  to  collect  and  disseminate  key  elements  of  a 
common  operational  picture  (COP)  to  maritime  forces 
and  on-scene  commanders  via  Link  16  and  Link  11 
datalink  systems  and  through  other  voice  communication 
and  data  transfer  systems.  However,  unresolved  tactical 
display,  communication  system,  and  data  transfer  system 
deficiencies  reduce  P-8ACOP  data  collection  and 
dissemination  capabilities  in  some  operational  scenarios. 
Corrections  for  these  deficiencies  will  be  evaluated  in 
future  operational  test  events. 

-  The  P-8A  is  operationally  suitable  for  Anti-Submarine 
Warfare  (ASW),  ASuW,  ISR,  C3,  and  related  maritime 
patrol  missions.  During  FOT&E,  P-8A  mission  capable 
rates  exceeded  those  measured  during  IOT&E.  Hardware 
components  continued  to  demonstrate  high  reliability 
during  operational  testing  and  system  software  reliability 
improved  compared  to  previous  IOT&E  results.  P-8A 
aircraft  flight  performance  (range,  speed,  payload,  etc.) 


meet  or  exceed  specified  operational  requirements  for 
ASW,  ASuW,  and  ISR  missions.  The  P-8A  meets  all 
ASuW  mission  endurance,  weapons  carriage,  and  mission 
turnaround  requirements  while  carrying  the  AGM-84D 
Block  1C  anti-ship  missile  system. 

-  The  P-8A  is  partially  compliant  with  joint  interoperability 
standards  and  Net  Ready  Key  Performance  Parameter 
requirements.  System  improvements  corrected  several 
high-priority  communication  system  interoperability  and 
net  ready  compliance  problems  identified  during  IOT&E. 
The  Joint  Interoperability  Test  Command  (JITC)  follow-up 
assessments  verified  compliance  with  all  39  specified 
mission  critical  information  exchange  requirements, 
leading  to  joint  use  certification  in  October  2013.  The 
JITC  assessment  also  identified  remaining  net  ready  and 
interoperability  shortfalls  that  require  future  improvement. 
The  Navy  has  identified  additional  system  enhancements 
necessary  to  deliver  a  fully  net-enabled  architecture  as 
principle  requirements  of  the  P-8A  Increment  3  upgrade 
program. 

•  As  part  of  the  P-8A  Increment  2  program,  the  Navy  is 
integrating  the  Multi-static  Active  Coherent  (MAC)  system 
into  the  P-8A  to  provide  the  initial  broad-area  ASW  search 
capabilities  originally  included  in  the  P-8A  Increment  1 
baseline  acquisition  program  and  supporting  operational 
requirement  documents.  DOT&E  previously  issued  a  MAC 
IOT&E  report  based  on  testing  conducted  on  the  P-3C 
aircraft.  This  report  concluded  that  the  MAC  system  provides 
the  P-3C  with  a  broad-area  ASW  search  capability  in  some 
operational  environments  and  for  select  target  scenarios,  but 
falls  short  of  fleet-defined  capabilities  needed  to  protect  high 
value  units.  Initial  test  results  indicate  that  MAC  performance 
varies  strongly  by  ocean  environmental  characteristics  and 
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target  tactics.  Further  information  can  be  found  in  DOT&E’s 
July  2014  IOT&E  report  on  the  MAC  System  on  P-3C 
Aircraft.  Until  a  fully-capable  broad-area  ASW  search  sensor 
is  successfully  integrated,  the  P-8A  will  be  unable  to  execute 
the  full  range  of  ASW  mission  tasks  defined  by  the  original 
P-8A  Increment  1  ASW  concept  of  operations. 

•  The  Navy  conducted  developmental  flight  testing  of  the  MAC 
Phase  I  System  broad-area  AS W  search  capability  on  the  P-8A 
in  February  through  March  2014  and  intended  to  commence 
operational  testing  in  May  2014.  Following  the  March  2014 
developmental  test  flights,  the  Navy  delayed  the  start  of 
operational  testing  to  correct  integration  and  performance 
deficiencies.  The  contractor  delivered  a  final  MAC  software 
update  in  August  2014.  Completion  of  developmental 
testing  and  start  of  operational  testing  is  scheduled  for 
November  2014. 

•  The  Navy  completed  the  first  lifetime  of  the  P-8A  full-scale 
structural  fatigue  testing  in  July  2014.  This  phase  of  testing 
identified  localized  fatigue  cracking  in  non-critical  structural 
components,  including  replaceable  pressure  web  and  aircraft 
tail  section  components.  The  program  is  currently  reviewing 
these  results  to  identify  fleet  airframe  inspection  requirements 
and  depot  repair  procedures  to  ensure  the  airframe  meets 

the  intended  25-year  design  life.  The  program  is  planning  a 
second  full  lifetime  of  structural  fatigue  testing  in  FY15  to 
complete  the  airframe  durability  assessment  and  to  further 
refine  inspection  and  repair  procedures. 

System 

•  The  P-8A  Poseidon  Multi-mission  Maritime  Aircraft  design 
is  based  on  the  Boeing  737-800  aircraft  with  significant 
modifications  to  support  Navy  maritime  patrol  mission 
requirements.  It  will  replace  the  P-3C  Orion. 

•  The  P-8A  incorporates  an  integrated  sensor  suite  that  includes 
radar,  electro-optical,  and  electronic  signal  detection  sensors 
to  detect,  identify,  locate,  and  track  surface  targets.  An 
integrated  acoustic  sonobuoy  launch  and  monitoring  system 
detects,  identifies,  locates,  and  tracks  submarine  targets.  The 


P-8A  carries  Mk  54  torpedoes  and  the  AGM-84D  Block  1C 
Harpoon  anti-ship  missile  system  to  engage  submarine  and 
maritime  surface  targets.  Sensor  systems  also  provide  tactical 
situational  awareness  information  for  dissemination  to  fleet 
forces  and  ISR  information  for  exploitation  by  the  joint 
intelligence  community. 

•  The  P-8A  aircraft  incorporates  aircraft  survivability 
enhancement  and  vulnerability  reduction  systems.  An 
integrated  infrared  (1R)  missile  detection  system,  flare 
dispenser,  and  directed  1R  countermeasure  system  is  designed 
to  improve  survivability  against  1R  missile  threats.  On  and 
off-board  sensors  and  datalink  systems  are  used  to  improve 
tactical  situational  awareness  of  expected  threat  systems. 

Fuel  tank  inerting  and  fire  protection  systems  reduce  aircraft 
vulnerability. 

•  The  Navy  is  integrating  the  MAC  system  into  the  P-8A  to 
provide  a  broad-area  active  ASW  search  capability. 

Mission 

•  Theater  Commanders  primarily  use  units  equipped  with 
the  P-8A  Multi-mission  Maritime  Aircraft  to  conduct  ASW 
operations  including  the  detection,  identification,  tracking,  and 
destruction  of  submarine  targets. 

•  Additional  P-8A  maritime  patrol  missions  include: 

-  ASuW  operations  to  detect,  identify,  track,  and  destroy 
enemy  surface  combatants  or  other  maritime  targets 

-  ISR  operations  to  collect  and  disseminate  imagery  and 
signals  information  for  exploitation  by  the  joint  intelligence 
community 

-  C3  operations  to  collect  and  disseminate  tactical  situation 
information  to  fleet  forces 

-  Identification  and  precise  geolocation  of  targets  ashore  to 
support  fleet  strike  warfare  missions 

Major  Contractor 

Boeing  Defense,  Space,  and  Security  -  St.  Louis,  Missouri 


Activity 

•  The  Navy  conducted  the  P-8A  Increment  1  FOT&E 
from  October  2013  through  March  2014  to  evaluate  the 
operational  effectiveness  and  suitability  of  the  integration  and 
employment  of  the  AGM-84D  Block  1C  Harpoon  anti-ship 
missile  system.  FOT&E  also  included  evaluation  of  software 
improvements  intended  to  correct  1 7  operationally  significant 
system  deficiencies  that  degraded  ASuW,  ISR,  and  C3  mission 
capabilities  or  system  survivability  during  IOT&E.  This 
FOT&E  did  not  re-evaluate  ASW  mission  capabilities  since  no 
ASW-related  system  improvements  were  included  in  the  flight 
software  version  assessed  during  this  test. 

-  FOT&E  included  15  flight  missions  totaling  76.2  hours  to 
evaluate  AGM-84D  operational  employment  effectiveness 


and  verify  correction  of  some  previously  identified  system 
deficiencies. 

-  DOT&E  evaluated  system  operational  suitability  during 
the  course  of  42  test  and  training  missions  totaling 
234.9  flight  hours  using  both  test  and  fleet  aircraft. 

-  The  Navy  conducted  integrated  test  and  dedicated  FOT&E 
events  at  Naval  Air  Station  (NAS)  Patuxent  River, 
Maryland;  Naval  Base  Ventura  County,  California;  and 
NAS  Jacksonville,  Florida. 

-  The  Navy  completed  all  testing  in  accordance  with  the 
DOT&E-approved  FOT&E  plan. 

•  The  Navy  completed  the  first  lifetime  of  the  P-8A  full-scale 

fatigue  and  durability  testing  in  FY14  using  the  fifth  P-8  A 
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production  aircraft  (YP-005).  Concurrent  “off-aircraft”  fatigue 
testing  of  selected  landing  gear  components  completed  the 
equivalent  of  two  design  lifetimes  of  testing.  Fatigue  test 
execution  and  data  analysis  activities  will  continue  into  FY15. 

•  The  Navy  continued  P-8A  Increment  2  software  development 
testing  in  FY14.  The  P-8A  Increment  2  program  integrates 
the  MAC  system  to  provide  a  limited  broad-area  AS  W  search 
capability  for  some  environments. 

-  The  Navy  conducted  developmental  flight  testing  of  the 
MAC  Phase  I  System  broad-area  ASW  search  capability 
on  the  P-8A  from  February  through  March  2014  to  support 
the  start  of  operational  testing  in  May  2014.  Following 
the  March  2014  test  flight,  the  Navy  delayed  the  start 

of  operational  testing  to  correct  MAC  integration  and 
performance  deficiencies. 

-  The  contractor  delivered  a  final  P-8A  MAC  software 
update  in  August  2014.  Completion  of  developmental 
testing  and  start  of  operational  testing  is  now  scheduled  for 
November  2014.  Future  Increment  2  program  upgrades 
include  new  high-altitude  ASW  capabilities  and  correction 
of  some  P-8A  deficiencies  identified  during  previous 
developmental  and  operational  test  events. 

Assessment 

•  The  P-8A  Increment  1  FOT&E  re-evaluated  armed  ASuW, 

ISR,  and  C3  mission  operational  effectiveness  and  operational 
suitability.  Based  on  FOT&E  results,  DOT&E  concluded  that: 

-  The  P-8A  Increment  1  system  provides  an  operationally 
effective  armed  ASuW  mission  capability  to  detect, 
classify,  and  track  maritime  surface  targets  and  engage 
them  using  the  AGM-84D  Block  1C  Harpoon  anti-ship 
missile.  During  FOT&E,  operational  crews  demonstrated 
the  capability  to  carry  and  employ  up  to  four  AGM-84D 
missiles  in  operationally  realistic  ASuW  mission 
scenarios.  System  software  changes  improved  surface 
target  search,  classification,  and  tracking  capabilities  as 
compared  to  observed  IOT&E  performance.  However, 
FOT&E  also  revealed  additional  sensor  integration  and 
display  deficiencies  that  increase  operator  workload  and/or 
preclude  use  of  some  ASuW  mission  system  capabilities. 

-  Based  on  FOT&E  results,  the  P-8  A  Increment  1  system 
does  not  yet  provide  an  operationally  effective  ISR  mission 
capability.  System  software  improvements  corrected 
several  high-priority  sensor  integration  problems  and 
improved  imagery  dissemination  capabilities.  However, 
persistent  SAR  image  quality  problems,  unresolved 
electronic  intelligence  collection  deficiencies,  and  data 
dissemination  limitations  continue  to  degrade  P-8A 

ISR  mission  capabilities.  Additional  SAR,  electronic 
intelligence  collection,  and  data  dissemination  upgrades  are 
scheduled  for  inclusion  in  upcoming  pre-planned  software 
updates  and  will  be  assessed  during  future  FOT&E  periods. 

-  The  P-8A  Increment  1  system  provides  a  C3  mission 
capability  to  collect  and  disseminate  key  elements 
of  a  common  operational  picture  (COP)  to  maritime 
forces  and  on-scene  commanders.  Operational  crews 


can  develop,  maintain,  and  disseminate  tactical  COP 
information  via  Link  16  and  Link  11  datalink  systems 
and  through  voice  communication  and  other  data  transfer 
systems.  However,  P-8A  C3  capabilities  are  reduced 
by  radar  and  threat  display  deficiencies  that  complicate 
COP  data  collection  in  high  traffic  operating  areas.  COP 
data  dissemination  capabilities  are  reduced  by  unreliable 
voice  satellite  communications,  Link  11  and  Link  16 
integration  problems,  Common  Data  Link  interoperability 
shortfalls,  and  a  limited  capability  to  exchange  intelligence 
information  via  satellite  communication  links.  Corrections 
for  these  deficiencies  will  be  evaluated  in  future 
operational  test  events. 

-  The  P-8A  is  operationally  suitable  for  ASW,  ASuW,  ISR, 
and  related  maritime  patrol  missions.  During  FOT&E, 
P-8A  mission  capable  rates  exceeded  those  measured 
during  IOT&E.  Hardware  components  continued  to 
demonstrate  high  reliability  during  operational  testing  and 
system  software  reliability  improved  compared  to  previous 
IOT&E  results.  P-8A  aircraft  flight  performance  (range, 
speed,  payload,  etc.)  meets  or  exceeds  specified  operational 
requirements  for  ASW,  ASuW,  and  ISR  missions.  The 
P-8A  meets  all  armed  ASuW  mission  endurance,  weapons 
carriage,  and  mission  turnaround  requirements  while 
carrying  the  AGM-84D  Block  1C  anti-ship  missile  system. 

-  The  P-8A  is  partially  compliant  with  joint  interoperability 
standards  and  Net  Ready  Key  Performance  Parameter 
requirements.  System  improvements  corrected  several 
high-priority  communication  system  interoperability  and 
net  ready  compliance  problems  identified  during  IOT&E. 
The  JITC  follow-up  assessments  verified  compliance 
with  all  39  specified  mission  critical  information 
exchange  requirements,  leading  to  joint  use  certification 
in  October  2013.  The  JITC  assessment  also  identified 
remaining  net  ready  and  interoperability  shortfalls  that 
require  future  improvement.  The  Navy  has  identified 
additional  system  enhancements  necessary  to  deliver  a 
fully  net-enabled  architecture  as  principle  requirements  of 
the  P-8A  Increment  3  upgrade  program. 

As  part  of  the  P-8A  Increment  2  program,  the  Navy  is 
integrating  the  MAC  system  into  the  P-8A  to  provide  the 
broad-area  ASW  search  capabilities  originally  included  in  the 
P-8A  Increment  1  baseline  acquisition  program  and  supporting 
operational  requirement  documents.  DOT&E  previously 
issued  a  MAC  IOT&E  report  based  on  testing  conducted  on 
the  P-3C  aircraft.  This  report  concluded  that  the  MAC  system 
provides  the  P-3C  with  a  broad-area  ASW  search  capability 
in  some  operational  environments  and  for  select  target 
scenarios,  but  falls  short  of  fleet-defined  capabilities  needed 
to  protect  high  value  units.  Initial  test  results  indicate  that 
MAC  performance  varies  strongly  by  ocean  environmental 
characteristics  and  target  tactics.  Further  information  can  be 
found  in  DOT&E’s  July  2014  IOT&E  report  on  the  MAC 
System  on  P-3C  Aircraft.  Until  a  fully-capable  broad-area 
ASW  search  sensor  is  successfully  integrated,  the  P-8A  will  be 
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unable  to  execute  the  full  range  of  AS  W  mission  tasks  defined 
by  the  original  P-8A  Increment  1  ASW  concept  of  operations. 

•  The  Navy  delayed  the  start  of  P-8A  MAC  OT&E  to  fix 
integration  and  performance  deficiencies  that  were  expected  to 
reduce  ASW  mission  performance  and  system  reliability.  The 
Navy’s  contractor  delivered  the  final  software  with  fixes  in 
August  2014.  Integration  laboratory  testing  and  flight  testing 
is  progressing  to  support  the  scheduled  start  of  operational 
testing  in  November  2014. 

•  The  Navy  completed  the  first  lifetime  of  the  P-8A  full-scale 
structural  fatigue  testing  in  July  2014.  This  phase  of  testing 
identified  localized  fatigue  cracking  in  non-critical  structural 
components,  including  replaceable  pressure  web  and  aircraft 
tail  section  components.  The  program  is  currently  reviewing 
these  results  to  identify  initial  fleet  airframe  inspection 
requirements  and  depot  repair  procedures  to  ensure  the 
airframe  meets  the  intended  25-year  design  life.  The  program 
is  planning  a  second  full  lifetime  of  structural  fatigue  testing 
in  FY15  to  complete  the  airframe  durability  assessment  and  to 
finalize  inspection  and  repair  procedures. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  made 
progress  on  all  four  FY13  recommendations.  The  Navy 
implemented  corrective  actions  and  conducted  FOT&E  for 


1 7  operationally  significant  system  deficiencies  related  to 
prioritized  ASuW,  ISR,  and  C3  mission  capabilities.  Future 
program  plans  include  additional  deficiency  correction 
activities  in  conjunction  with  P-8  A  Increment  2  development 
and  testing.  DOT&E  approved  the  operational  test  plans  for 
the  P-8A  Increment  2  program  designed  to  evaluate  previously 
deferred  ASW  broad-area  search  capabilities.  The  Navy 
is  also  reviewing  and  prioritizing  potential  future  system 
upgrades,  including  consideration  of  options  to  integrate 
RF  threat  warning  systems  and  improve  the  Dry  Bay  Fire 
Protection  System,  as  previously  recommended  by  DOT&E. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Continue  to  implement  corrective  actions  for  significant 
system  deficiencies  identified  in  previous  P-8A 
developmental  and  operational  test  reports  and  conduct 
additional  follow-on  operational  tests  to  verify  improved 
mission  capabilities. 

2.  Complete  adequate  operational  testing  of  deferred  ASW 
broad-area  search  capabilities  in  conjunction  with  the  P-8  A 
Increment  2  program. 

3.  Plan  and  conduct  operational  testing  of  new  P-8  A  system 
improvements  intended  to  provide  high-altitude  ASW 
search  and  attack  capabilities  in  conjunction  with  the  P-8A 
Increment  2  and  Increment  3  programs. 
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Remote  Minehunting  System  (RMS) 


Executive  Summary 

•  In  June  2014,  DOT&E  reported  the  Remote  Minehunting 
System  (RMS)  (consisting  of  a  version  4.2  (v4.2)  Remote 
Multi-Mission  Vehicle  (RMMV)  and  AN/AQS-20A  sonar) 
had  not  demonstrated  sufficient  performance  or  successful 
integration  with  interfacing  Littoral  Combat  Ship  (LCS) 
systems  to  demonstrate  the  Navy’s  minimum  Increment  1 
warfighting  capability.  Although  the  Navy  is  working 

on  upgrades  to  improve  system  performance  and  LCS 
capability  in  the  v6.0  RMMV  and  the  AN/AQS-20A/B  sonar, 
developmental  testing  completed  in  1QFY15  demonstrated 
continued  performance  issues  and  RMS  mission  package 
integration  challenges. 

•  In  4QFY14,  USD(AT&L)  delayed  a  review  to  consider 
approval  of  RMS  low-rate  initial  production  (LRIP)  until 
3QFY15. 

•  Combined  developmental  and  integrated  testing  completed 
in  FY14  provides  a  point  estimate  for  v4.2  vehicle  reliability 
of  31.3  hours  Mean  Time  Between  Operational  Mission 
Failure  (MTBOMF).  Developmental  testing  completed  in 
1QFY 15  provides  a  point  estimate  for  v6.0  vehicle  reliability 
of  34.6  hours  MTBOMF.  Statistical  analysis  of  all  test  data 
indicates  the  result  is  not  sufficient  to  conclude  that  reliability 
has  actually  improved  since  a  Nunn-McC  urdy  review  of  the 
program  in  2010.  Therefore,  test  data  currently  available 
(including  early  testing  of  the  v6.0  vehicle)  do  not  support 
the  Navy’s  assertion  that  vehicle  reliability  has  improved. 
Moreover,  the  current  estimate  of  RMS  reliability,  once  all 
of  the  other  components  of  the  system  are  considered,  is  no 
more  than  20  hours  MTBOMF,  which  is  well-short  of  what 

is  needed  to  complete  MCM  missions  in  a  timely  fashion  and 
meet  the  Navy’s  desired  mission  timelines. 

•  Developmental  testing  conducted  in  FY14  and  1QFY15 
continued  to  show  that  the  AN/AQS-20A  sonar  does  not 
meet  all  Navy  requirements.  The  Navy  expected  to  correct 
these  deficiencies  prior  to  operational  testing  in  FY15  by 
implementing  pre-planned  product  improvements  (the 
AN/AQS-20B  version  of  the  sonar)  and  integrating  the 
improved  sensor  into  the  Mine  Countermeasures  (MCM) 
mission  package.  Delays  in  the  delivery  of  AN/AQS-20B 
prototypes  and  problems  discovered  in  early  characterization 
testing  in  FY14  leave  little  time  to  complete  necessary 
developmental  and  operational  testing  of  the  AN/AQS-20B 
prior  to  the  planned  operational  testing  of  LCS  equipped  with 
the  MCM  mission  package  in  FY15. 

•  Communications  ranges  afforded  by  current  RMS  radios 
will  require  operational  commanders  to  clear  a  series  of 
LCS  operating  boxes  to  support  minehunting  and  clearance 
operations.  These  operating  boxes  will  be  necessary  to  keep 
an  LCS  and  its  crew  out  of  the  minefield  while  operating 
the  RMS  in  searches  for  mine-like  objects  or  identifying 


bottom  objects  located  within  shipping  lanes  that  are  longer 
than  demonstrated  communications  ranges.  Additional 
effort  to  clear  operating  boxes  will  increase  the  demand  for 
mine  clearance  and  delay  attainment  of  strategic  objectives. 
The  analysis  of  communications  data  collected  during  the 
most  recent  period  of  LCS  developmental  testing  is  still  in 
progress,  but  test  observers  reported  continued  communication 
problems. 

System 

•  The  RMS  is  designed  to  provide  an  organic,  off-board  mine 
reconnaissance  capability  to  detect,  classify,  and  localize 
non-buried  bottom  and  moored  mines,  as  well  as  to  identify 
shallow-water  bottom  mines  only. 

•  The  RMS  will  be  launched,  operated,  and  recovered  from  the 
LCS  as  part  of  the  MCM  mission  package  (when  embarked). 

•  The  RMS  is  comprised  of  four  major  components: 

-  RMMV 

■  The  RMMV  is  an  unmanned,  semi-submersible, 
untethered  vehicle  designed  to  conduct  autonomous  or 
semi-autonomous  mine  reconnaissance  missions. 

■  The  RMMV  physically  transports  AN/AQS-20A/B 
sensors,  processors,  and  datalink  equipment  to  the 
operations  area  where  mine  reconnaissance  data  are 
collected,  recorded,  and  transmitted  to  the  host  LCS 
platform. 

■  RMS  sensor  data  are  recorded  to  a  removable  hard  drive 
during  minehunting  operations.  Following  vehicle 
recovery,  operators  transfer  data  to  an  Organic  Post 
Mission  Analysis  station  and  review  sonar  data  to  mark 
contacts  as  suspected  mine-like  objects. 
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■  The  Navy  has  not  yet  presented  its  plans  to  incorporate 
reliability  growth  improvements  in  a  future  system 
increment  (LRIP  2). 

-  AN/AQS-20A  sensor 

■  The  AN/AQS-20A  is  a  variable  depth  forward-looking 
and  side-scanning  sonar  that  is  deployed  and  retrieved  by 
the  RMMV. 

■  The  sensor  tow  body  automatically  controls  depth  based 
on  specific  mission  planning  parameters,  providing  a 
stable  platform  for  integral  mine  reconnaissance  sensors. 

■  The  AN/AQS-20A  provides  detection,  classification,  and 
localization  of  non-buried  bottom  (on  the  ocean  floor) 
and  volume  (in  the  water  column)  mine-like-contacts. 
The  sensor  utilizes  port  and  starboard  Side-Looking 
Sonars  and  a  Gap  Filler  Sonar  for  detection  of  bottom 
and  closely  tethered  moored  mines  in  shallow  waters. 

A  Volume  Search  Sonar  (VSS)  and  a  Forward-Looking 
Sonar  are  utilized  for  all  other  moored  mine  detection. 

An  Electro-Optic  Identification  Device  can  replace  the 
VSS  for  missions  requiring  identification  (mine  versus 
non-mine)  of  shallow-water  bottom  mine-like  contacts 
via  high-resolution  imaging. 

■  The  Navy  plans  to  incorporate  an  improved 
forward-looking  sonar  and  new  synthetic  aperture 
side-looking  sonars  (AN/AQS-20B)  in  FY15. 

-  Remote  Minehunting  Functional  Segment  (RMFS) 

■  The  RMFS  is  the  control  system  hosted  in  the  mission 
package  computing  environment  on  the  LCS. 

■  RMFS  is  a  two-operator  system  that  enables  the  Remote 
Vehicle  Operator  and  Remote  Sensor  Operator  to 
command  and  monitor  RMS  operations. 

■  Specific  RMFS  functionality  enables  the  operator  to 
(1)  command  and  monitor  the  RMMV;  (2)  receive, 
process,  and  display  real-time  mission  data;  (3)  conduct 


performance  monitoring/fault  detection/fault  localization; 
and  (4)  perform  network  communication  to  the  Data  Link 
System  (DLS). 

■  RMFS  also  exchanges  data  with  the  Global  Command 
and  Control  System  -  Maritime/Mine  Warfare 
Environmental  Decision  Aids  Library  for  mission 
planning  and  interface  to  the  Department  of  Defense 
Information  Network. 

-  DLS 

■  The  DLS  enables  the  RMMV  to  communicate  with 
the  LCS  MCM  mission  package  via  one  of  two  radio 
frequency  datalink  subsystems. 

■  The  Multi- Vehicle  Communications  System  (MVCS) 
consists  of  two  radios  -  an  Ultra  High  Frequency 
line-of-sight  datalink  that  is  used  for  vehicle  launch  and 
recovery  and  near-ship  operations  and  a  low-band  Very 
High  Frequency  datalink  that  is  used  for  over-the-horizon 
mission  operations.  Both  datalinks  provide  encrypted 
communications  between  the  LCS  MCM  mission 
package  and  the  RMMV  for  real-time  command  and 
control  and  transmission  of  some  sensor  data. 

Mission 

MCM  Commanders  will  employ  the  RMS  from  an  MCM 
mission  package-equipped  LCS,  to  detect,  classify,  and  localize 
non-buried  bottom  and  moored  mines,  as  well  as  to  identify 
shallow-water  bottom  mines  in  support  of  theater  minehunting 
operations  in  shallow-water  and  deep-water  minefields. 

Major  Contractors 

•  RMMV:  Lockheed  Martin  -  West  Palm  Beach,  Florida 

•  AN/AQS-20A:  Raytheon  Corporation  -  Portsmouth,  Rhode 
Island 


Activity 

•  In  1QFY14,  the  Navy  completed  two  phases  of  developmental 
testing  (DT-IIG)  of  the  RMS  consisting  of  a  v4.2  RMMV  and 
an  AN/AQS-20A  sensor  from  a  shore  base  at  the  contractor’s 
facility  in  West  Palm  Beach,  Florida.  A  third  phase  of  testing 
described  by  the  RMS  Test  and  Evaluation  Master  Plan 
(TEMP)  as  an  opportunity  to  assess  risk  of  the  interfaces 
with  the  LCS,  including  cybersecurity,  was  not  conducted. 

The  Navy  cited  lack  of  LCS  availability  as  the  rationale  for 
cancellation  of  this  phase,  but  incompatibility  of  the  v4.2 
RMMV  with  LCS  was  also  a  factor.  The  v4.2  vehicles  did  not 
have  the  needed  structural  improvements  or  communications 
upgrades  to  make  them  compatible  with  LCS  systems. 

•  DOT&E  disapproved  the  Navy’s  plan  to  conduct  an 
operational  assessment  (OA)  of  the  RMS  in  2QFY14  because 
the  OA  would  have  been  a  wasted  effort  for  the  following 
reasons: 

-  The  proposed  test  article  was  not  representative  of  the 
system  the  Navy  plans  to  employ  in  the  first  increment  of 
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the  LCS  MCM  mission  package  and  therefore  would  not 
provide  data  necessary  to  augment  the  operational  testing 
of  an  LCS  equipped  with  that  mission  package. 

-  Test  limitations  would  have  precluded  an  operational 
evaluation  of  key  phases  of  the  end-to-end  mission. 

-  Conduct  of  the  test  would  have  delayed  vehicle  upgrades 
necessary  to  support  testing  of  the  system  the  Navy  expects 
to  field. 

•  DOT&E  advised  the  Navy  that  this  testing  may  be  conducted 
as  combined  developmental  and  integrated  testing  if  the 
standards  of  operational  realism  are  met.  The  Navy  expects 
to  conduct  testing  from  December  2014  to  January  2015  to 
satisfy  objectives  of  the  planned  OA. 

•  The  Navy  initiated  updates  to  the  RMS  and  AN/AQS-20A 
TEMPs  that  merged  the  TEMPs  into  a  single  document 

in  2QFY14.  To  avoid  confusion  and  promote  efficiency, 
DOT&E  advised  the  Navy  that  both  TEMPs  should  be  further 
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combined  in  the  LCS  TEMP,  which  is  also  being  updated.  It  is 
unclear  when  updates  to  either  the  RMS  or  LCS  TEMPs  will 
be  completed. 

•  The  Navy  continued  to  develop  pre-planned  product 
improvements  for  the  AN/AQS-20A  and  is  investigating 
improved  tactics,  techniques,  and  procedures  for  its 
employment.  Both  efforts  are  intended  to  mitigate  deficiencies 
observed  during  previous  OAs  and  developmental  testing 

of  the  RMS  and  AN/AQS-20A.  The  Navy  plans  to  conduct 
developmental  testing  of  the  RMS  configured  with  the  newly 
integrated  v6.0  RMMV  and  the  improved  AN/AQS-20B 
sensor  in  late  lQ/early  2QFY15. 

•  The  Navy  imposed  a  weight  limit  on  LCS  2  and  LCS  4  launch 
and  recovery  systems  as  a  result  of  finding  cracks  in  the  welds 
at  the  base  of  support  stanchions  located  in  the  mission  bay 
that  the  Navy  attributes  to  LCS  2’s  heavy  weather  testing. 

As  a  result,  subsequent  testing  of  the  launch  and  recovery 
operations  could  be  conducted  only  when  wave-induced 
loading  on  the  recovery  system  (a  function  of  wave  height  and 
period)  did  not  exceed  32,000  pounds-force.  (For  example,  a 
significant  wave  height  of  2  feet  coupled  with  a  wave  period 
of  2  seconds,  which  could  be  encountered  in  a  Sea  State  2, 
would  preclude  RMMV  recovery  until  calmer  sea  conditions 
developed.)  The  Navy  reported  they  are  making  design 
changes  to  LCS  6  and  later  seaframes  to  correct  the  stress 
cracking  problem  and  remove  the  launch  and  recovery  system 
weight  limit.  LCS  2  and  LCS  4  will  be  corrected  during  their 
next  shipyard  availabilities.  Prior  to  discovering  the  damaged 
support  stanchions,  the  Navy  had  already  imposed  a  sea  state 
limit  of  less  than  Sea  State  3  because  of  the  erratic  motion  of 
the  RMMV  in  the  ship’s  wake,  which  had  caused  damage  to 
capture  spines  and  RMMVs. 

•  In  3QFY14,  the  Navy  conducted  dockside  and  at-sea 
developmental  testing  to  verify  correction  of  RMMV 
launch,  handling,  and  recovery  system  and  communications 
deficiencies  observed  in  FY13  developmental  testing. 

•  DOT&E  provided  an  assessment  of  RMS  performance  in 
testing  to  members  of  the  Defense  Acquisition  Board  in 

June  2014  after  the  program  was  recertified  following  an  FY10 
Nunn-McCurdy  breach.  In  4QFY 14,  USD(AT&L)  delayed  a 
review  to  consider  approval  of  RMS  LRIP  until  FY15. 

•  The  Navy  commenced  additional  dockside  and  at-sea 
developmental  testing  of  the  RMMV  launch,  handling,  and 
recovery  system  and  the  multi-vehicle  communications 
system  in  4QFY14.  In  1QFY15,  the  Navy  conducted  the  last 
scheduled  phase  of  the  Increment  1  MCM  mission  package 
developmental  test  DT-B2  aboard  USS  Independence  (LCS  2). 
This  phase  was  the  first  time  that  RMS  (configured  with  a  v6.0 
RMMV)  and  the  airborne  MCM  mission  package  components 
had  operated  together  off  an  LCS.  Flowever,  because  the  Navy 
now  expects  to  make  additional  RMMV  software  changes 
and  deferred  a  decision  on  which  variant  of  the  AN/AQS-20 

to  field  until  December  2014,  LCS  Technical  Evaluation  is 
expected  to  be  the  first  time  the  fielded  system  will  be  tested  in 
realistic  end-to-end  missions. 


•  The  Joint  Requirements  Oversight  Council  (JROC)  and  the 
Navy  approved  the  RMS  Capability  Production  Document  in 
March  2014. 

Assessment 

•  As  DOT&E  reported  to  members  of  the  Defense  Acquisition 
Board  in  June  2014,  the  combined  results  of  shore-based  and 
LCS-based  testing  conducted  since  the  program  was  recertified 
following  a  Nunn-McCurdy  in  2010  have  not  demonstrated 
that  an  LCS  equipped  with  an  MCM  mission  package  that 
includes  two  RMMVs  and  three  AN/AQS-20A  sonars  will  be 
able  to  support  the  sustained  area  coverage  rate  that  the  Navy 
has  established  for  the  Increment  1  MCM  mission  package. 
Few  test  data  are  available  to  indicate  whether  planned  RMS 
improvements  would  support  meeting  the  Increment  1  LCS 
requirements,  let  alone  provide  the  more  robust  capability 

the  Navy  expects  to  achieve  with  Increment  4  of  the  MCM 
mission  package. 

•  The  requirements,  in  the  approved  RMS  Capability 
Development  Document  and  Capability  Production  Document, 
are  not  consistent  with  the  approved  requirements  for 
minehunting  conducted  by  the  LCS. 

-  The  RMS  could  meet  all  threshold  requirements  designated 
as  Key  Performance  Parameters  and  Key  System  Attributes 
and  still  not  support  the  LCS  MCM  mission  because  the 
RMS  requirements  lack  the  appropriate  mission  focus. 

-  The  RMS  search  rate  requirement,  for  example,  has 
no  limit  on  false  alarms  and  excludes  time  required 
for  planning,  transit  to  and  from  the  search  area, 
operator  assessment,  follow-on  actions  to  reduce  false 
classifications,  and  efforts  to  recover  from  failures.  All  of 
these  factors  directly  affect  the  achievement  of  the  required 
performance  and  timeliness  of  LCS  MCM  missions. 

-  The  reliability  of  the  v4.2  RMMV  during  combined 
developmental  and  integrated  testing  completed  in  FY14 
was  31.3  hours  MTBOMF,  which  is  well  below  the 
required  reliability  of  75  hours  MTBOMF.  Although  the 
Program  Office  maintains  that  the  RMMV  reliability  is 
substantially  above  that  value  and  that  a  reliability  growth 
program  completed  in  FY13  was  highly  successful,  the 
Navy’s  reliability  analysis  is  fundamentally  flawed  because 
it  overstates  RMMV  operating  time  and  undercounts  the 
number  of  operational  mission  failures.  The  RMMV 
reliability  issue  has  been  the  principal  reason  that  the 
program  has  not  attempted  to  reenter  operational  test. 

-  The  vendor  completed  an  RMMV  growth  program  and 
has  subsequently  incorporated  additional  fixes  to  correct 
deficiencies  observed  during  developmental  testing  and 
combined  developmental/integrated  testing  in  2013. 

-  The  restoration  from  RMS  failures/faults  when  operating 
from  an  LCS  is  reliant  on  intermediate-  and  depot-level 
(off-board)  maintenance  support.  Organizational-level 
(shipboard)  maintenance  support  to  restore  system 
availability  necessary  to  complete  timely  and  realistic 
operational  scenarios  is  limited. 
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-  The  system’s  AN/AQS-20A  sensor  also  has  reliability 
problems. 

•  The  results  of  combined  developmental  and  integrated  testing 
completed  in  FY14  continued  to  show  that  the  AN/AQS-20A 
sensor  does  not  meet  Navy  requirements  for  contact  depth 
localization  accuracy  or  false  classification  density  (number  of 
contacts  erroneously  classified  as  mine-like  objects  per  unit  of 
area  searched). 

-  Contact  depth  (vertical  localization)  errors  and  false 
classification  density  exceeded  Navy  limits  in  all  AQS-20A 
operating  modes. 

-  The  sensor  also  has  trouble  meeting  the  probability  of 
detection  and  classification  requirement  in  shallow  waters 
and  RMS  has  difficulty  guiding  the  sensor  over  bottom 
contacts  for  identification  in  deep  water. 

-  Although  the  Navy  is  working  on  pre-planned  product 
improvements  in  the  AN/AQS-20B  version  of  the  system, 
no  test  data  are  available  to  indicate  that  problems 

with  false  classifications  and  vertical  localization 
errors  have  been  mitigated.  Delays  in  the  delivery  of 
AN/AQS-20B  prototypes  and  problems  discovered  in 
early  characterization  testing  in  FY14  leave  little  time  to 
complete  necessary  developmental  and  operational  testing 
of  the  AN/AQS-20B  prior  to  the  planned  operational 
testing  of  LCS  equipped  with  the  MCM  mission  package 
in  FY15.  If  left  uncorrected,  AN/AQS-20A/B  problems 
will  reduce  the  minehunting  and  clearing  capability  of  the 
MCM  mission  package  and  the  LCS  will  not  meet  interim 
area  clearance  requirements  in  ideal  conditions  let  alone 
more  realistic  area  clearance  needs  for  the  threat  scenarios 
the  mission  package  was  developed  to  counter  in  theater. 

•  The  RMS  program  has  not  yet  demonstrated  that  the  system 
can  meet  its  detection  and  classification  requirements 
against  moored  and  bottom  mines  spanning  the  portion  of 
the  shallow  water  regime  not  covered  by  the  Airborne  Laser 
Mine  Detection  System;  the  program  anticipates  that  the 
AN/AQS-20B  sensor  will  permit  this  capability.  If  this  cannot 
be  accomplished,  the  Navy  will  need  to  conduct  additional 
search  passes  to  achieve  adequate  coverage  of  the  water 
column,  resulting  in  increased  detection  and  classification 
timelines  and  the  LCS  not  meeting  area  clearance  rate 
requirements. 

•  Testing  completed  in  1QFY14  indicates  that  fleet  operators 
frequently  misclassify  moored  mine  targets  as  bottom  objects 
during  shallow  water  minehunting.  These  errors  can  increase 
the  difficulty  of  reacquiring  mines  during  the  neutralization 
phase  of  MCM  operations.  The  Navy  is  weighing  the  need  for 
additional  search  passes  to  resolve  mine  position  uncertainty 
before  proceeding  to  mine  clearance  operations.  Such  tactics 
woidd  require  more  time  to  accomplish,  resulting  in  increased 
mine  clearance  timelines  and  the  LCS  not  meeting  area 
clearance  rate  requirements. 

•  RMS  radios  have  had  difficulty  establishing  reliable 
communications  with  the  LCS  during  developmental  testing, 
and  once  communications  are  established,  the  current 
communications  systems  do  not  support  RMMV  mine 


identification  operations  beyond  the  horizon.  Although  the 
RMMV  can  search  autonomously  while  operating  over  the 
horizon  from  the  LCS,  it  currently  only  can  conduct  operations 
to  reacquire  and  identify  bottom  mines  within  the  range  of 
Ultra  High  Frequency  communications.  This  limitation  will 
complicate  MCM  operations  in  long  shipping  channels,  and 
may  make  it  necessary  to  clear  a  series  of  LCS  operating 
areas  to  allow  MCM  operations  to  progress  along  the  channel. 
The  cleared  operating  areas  will  be  needed  to  keep  the  LCS 
and  its  crew  out  of  mined  waters.  The  additional  effort 
required  to  clear  these  LCS  operating  areas  would  increase  the 
demand  for  mine  clearance  and  delay  attainment  of  strategic 
objectives.  This  issue  is  not  new  to  RMS;  however,  it  did  not 
become  operationally  significant  until  the  Navy  decertified 
the  MH-60S  helicopter  for  towing  MCM  devices,  including 
the  AN/AQS-20A/B  sensor.  The  RMS  communication  range 
limitation  was  not  an  operational  concern  when  the  option 
existed  for  the  helicopter  with  towed  sensor  to  conduct 
identification  operations  beyond  the  horizon.  The  Navy  has 
not  yet  identified  a  solution.  The  analysis  of  communications 
data  collected  during  the  most  recent  period  of  LCS 
developmental  testing  is  still  in  progress,  but  test  observers 
reported  continued  communications  problems. 

•  The  Independence  class  LCS  has  had  difficulty  launching  and 
recovering  the  RMMV  because  of  the  vehicle’s  erratic  motion 
in  the  ship’s  wake.  In  past  developmental  testing,  violent 
RMMV  yaw  and  roll  motions  have  overstressed  and  damaged 
the  launch  and  recovery  hardware  and  resulted  in  damage  to 
the  RMMV,  which  led  to  the  Navy  imposing  a  Sea  State  3 
limit  on  handling  operations.  Following  changes  to  launch 
and  recovery  hardware,  procedures,  training,  and  RMMV 
hardware,  the  Navy  demonstrated  16  RMMV  launches  and 
14  RMMV  recoveries  during  23  days  at  sea  in  developmental 
testing  during  favorable  sea  state  conditions  and  within  the 
imposed  weight  loading  restrictions  in  1QFY15.  Nonetheless, 
the  most  recent  period  of  developmental  testing  witnessed 
several  instances  of  equipment  damage  that  delayed  or 
prevented  recovery  of  an  off-board  RMMV. 

•  Following  the  October  2014  phase  of  developmental  testing, 
the  Navy  identified  a  new  problem  with  the  redesigned 
lifting  structure  used  in  the  vehicle’s  launch  and  recovery. 
Deformation  in  the  capture  probe  and  corresponding  latching 
mechanism  in  the  vehicle  (capture  pawls)  was  observed 
following  several  failed  attempts  to  recover  the  RMMV. 

This  is  significant  because  the  entire  weight  of  the  vehicle  is 
supported  by  these  components  during  vehicle  launch  and 
recovery,  and  the  defects  pose  a  safety  concern  for  mission 
package  personnel  who  must  work  in  close  proximity  to 

the  suspended  RMMV  to  secure  it  to  the  cradle  pallet  in  the 
mission  bay.  The  Navy  identified  substandard  metallurgical 
strengthening  as  the  root  cause  of  observed  deformation.  In 
addition,  non-load  bearing  components  of  the  redesigned 
RMMV  capture  spine  assemblies  experienced  multiple  failures 
including  several  the  test  team  attributed  to  substandard 
welds.  In  some  cases,  the  team  test  was  unable  to  continue 
RMS  operations  without  replacement  parts  from  shore,  which 
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in  theater  would  preclude  sustained  RMS  operations  without 
excessive  reliance  on  shore-based  support.  The  Navy  must 
correct  capture  spine  deficiencies  to  ensure  safe  and  sustained 
RMMV  launch  and  recovery  in  support  of  LCS  MCM 
operations. 

•  Developmental  testing  conducted  aboard  LCS  2  in  1QFY15 
also  indicates  that  many  RMS  performance  problems  identified 
in  earlier  test  phases  have  not  been  corrected  and,  in  some 
cases,  new  problems  have  been  introduced  following  changes 
to  system  configurations  or  tactics. 

-  RMS  reliability  problems  persisted  in  the  recent  phase 
of  developmental  testing  (1QFY15)  evidenced  in  part  by 
fewer  vehicle  recoveries  than  vehicle  launches.  Problems 
observed  include  the  inability  to  align  the  system’s  inertial 
navigational  unit,  intermittent  communications,  a  lube  oil 
pump  failure  that  caused  a  mission  abort,  capture  latch 
impairment  that  precluded  shipboard  recovery  of  the 
RMMV,  degraded  electro-optic  identification  resulting 

in  a  mission  abort  to  replace  the  AN/AQS-20A  towed 
body,  tow  cable  damage  following  an  apparent  snag  that 
rendered  the  system  inoperable  in  the  assigned  mission 
until  a  replacement  tow  cable  could  be  installed  with  the 
assistance  of  shore-based  support,  and  multiple  incidents 
of  AN/AQS-20A  stuck  fins  or  fin  actuation  faults. 

Although  the  Navy  demonstrated  more  frequent  RMMV 
launches  during  this  period  of  testing,  continued  RMS 
reliability  problems  limited  system  minehunting  to  less 
than  50  hours  during  the  three  weeks  of  most  intensive 
testing  (approximately  16  hours  per  week).  LCS  reliability 
problems  also  forced  the  ship  to  remain  in  port  for  repairs 
instead  of  conducting  at-sea  RMS  testing  as  planned. 
Including  an  additional  week  spent  in  port  for  LCS  repairs, 
RMS  averaged  approximately  12  hours  of  minehunting 
per  week.  This  result  is  consistent  with  the  assessment 
of  RMS  capability  DOT&E  provided  to  members  of  the 
DAB  following  RMMV  v4.2  and  AN/AQS-20A  testing 
to  indicate  that  the  Navy  had  not  yet  demonstrated  that  it 
could  sustain  operations  of  more  than  one  14-hour  RMMV 
sortie  per  week  (i.e.,  10  to  12  hours  of  RMS  minehunting 
per  week).  Unless  greater  minehunting  operating  tempo  is 
achieved  the  Navy  will  not  meet  its  interim  area  clearance 
rate  requirements. 

-  The  Navy  reported  that  the  RMS  operated  for 
approximately  140  hours  during  LCS  developmental 
testing  in  1QFY15.  DOT&E ’s  preliminary  assessment 
of  test  data  identified  at  least  seven  RMS  failures  that 
precluded  vehicle  recovery,  required  sensor  replacement, 
or  required  assistance  from  shore -based  support  contractors 
to  restore  system  availability.  In  operational  testing,  these 
failures  would  be  assessed  as  operational  mission  failures. 
Thus,  by  operational  criteria,  RMS  demonstrated  reliability 
was  no  more  than  20  hours  MTBOMF  during  this  phase 
of  testing.  Because  much  of  the  operating  time  cited  by 
the  Navy  was  not  devoted  to  minehunting  activities,  this 
estimate  should  be  considered  an  upper  bound  for  current 
RMS  operational  reliability.  Moreover,  statistical  analysis 


of  all  existing  data  do  not  yet  support  the  Navy’s  assertions 
of  improving  RMS  reliability. 

-  As  in  previous  testing,  fleet  operators  were  unable 
to  execute  operationally-realistic,  end-to-end  mine 
reconnaissance  and  clearance  without  the  assistance  of 
testers  with  knowledge  of  ground  truth  target  positions. 
Using  mission  package  tools  such  as  Organic  Post 
Mission  Analysis  and  the  new  Contact  Management  Tool 
(CMT)  fleet  operators  failed  to  convey  some  mine  targets 
correctly  detected  by  the  RMS  in  an  initial  search  pass 

to  the  Airborne  Mine  Neutralization  System  (AMNS)  for 
neutralization.  The  Navy  continues  to  investigate  the  root 
cause  of  incorrectly  dropped  contacts,  which  will  severely 
limit  LCS  MCM  mission  effectiveness  unless  corrected. 

-  Multiple-pass  tactics  and  the  CMT  were  introduced  to 
minimize  the  number  of  false  classifications  passed  on  for 
identification/neutralization.  Flowever,  during  the  recent 
phase  of  testing,  the  number  of  in-volume  RMS  false 
classifications  remaining  in  the  contact  list  after  multiple 
RMS  search  passes  and  contact  correlation  far  exceeded 
Navy  pre-test  predictions  indicating  these  tactics  and  tools 
have  not  yet  achieved  the  desired  result.  The  CMT  also 
failed  during  the  later  stages  of  testing  requiring  operators 
to  attempt  the  cumbersome  task  of  manually  tracking 
contacts  between  stages  of  the  RMS  mission  before 
determining  which  contacts  merited  further  investigation. 

If  large  numbers  of  RMS  in-volume  false  classifications 
are  passed  to  the  AMNS  for  follow-on  action,  LCS  mine 
clearance  requirements  will  not  be  met.  Large  quantities 
of  contacts  generated  by  the  RMS  make  manual  correlation 
time  consuming  and  increase  the  potential  to  drop  correct 
classifications;  both  of  these  problems  will  limit  LCS 
MCM  effectiveness  unless  corrected. 

•  The  Navy  has  not  assessed  radiated  noise  following  recent 
vehicle  configuration  changes.  Radiated  noise  measurements 
were  last  collected  during  developmental  testing  in  2007/2008, 
and  indicated  that  existing  RMMVs  may  be  vulnerable  to 
some  mines. 

•  The  RMS  program  has  not  conducted  operational  testing 
of  cybersecurity  of  the  RMS  hardware  and  software 
configurations  intended  for  Initial  Operational  Capability  in 
the  LCS  MCM  mission  package  in  FY15/16. 

•  No  RMMV  launch  and  recovery  operations  have  been 
conducted  aboard  a  Freedom  class  LCS  at  sea. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  made 
progress  on  all  four  FY13  recommendations.  Shore -based 
testing  completed  in  1QFY14  and  shipboard  testing  completed 
in  1QFY15  provided  additional  information  regarding  RMS, 
RMMV,  and  AN/AQS-20A  reliability;  RMS  operational 
availability;  and  RMMV  launch,  handling,  and  recovery 
system  performance.  Although  the  Navy  continues  to  develop 
and  test  AN/AQS-20  upgrades,  it  has  not  demonstrated  in 
developmental  or  operational  testing  that  it  has  corrected 
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problems  with  false  classifications  and  contact  localization 
errors  that  will  otherwise  limit  performance  in  the  planned 
FY15  operational  testing.  The  Navy  expects  to  complete 
its  update  to  the  RMS  TEMP,  which  now  includes  the 
AN/AQS-20  sonar,  by  2QFY15. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Identify  the  RMS  configuration  for  operational  testing  of 
LCS  equipped  with  the  first  increment  of  MCM  capability 
and  complete  the  required  operationally  realistic  testing  of 
that  system  prior  to  LCS  MCM  mission  package  Technical 
Evaluation. 

2.  Conduct  testing  of  the  RMS  consisting  of  the  v6.0  RMMV 
and  AN/AQS-20B  in  operationally  realistic  end-to-end 
minehunting  missions  to  characterize  AN/AQS-20B 
minehunting  performance  and  accurately  assess 
availability  of  the  RMS  and  reliability  of  the  RMMV  and 
AN/AQS-20B. 

3.  Investigate  the  use  of  communications  relays  and  other 
solutions  that  might  improve  the  standoff  distance  between 
an  RMMV  and  its  host  ship  to  improve  the  efficiency  of 
LCS  MCM  operations. 

4.  Conduct  cybersecurity  testing  of  the  RMS  to  identify  risks 
associated  with  deploying  the  system  planned  for  Initial 
Operational  Capability  in  FY15/16. 

5.  Document  a  robust  reliability  monitoring  and  growth 
strategy  for  any  new  LRIP  vehicles  procured  following  a 
planned  FY15  Milestone  C  decision. 

6.  Reassess  RMMV  v6.0  radiated  noise  following  vehicle 
upgrades. 


7.  Reexamine  minimum  vehicle  and  sensor  reliability  and 
LCS  organizational-level  maintenance  support  needed  to 
complete  timely  and  realistic  operational  scenarios  without 
excessive  reliance  on  intermediate-  and  depot-level  support. 

8.  Reconsider  RMS  minehunting  requirements  in  the  context 
of  expected  LCS  tactics  and  operations. 

9.  Recognizing  schedule  constraints,  continue  to  conduct 
ship-based  testing  of  the  RMS  that  includes  end-to-end 
minehunting  missions  from  an  LCS  as  part  of  the  MCM 
mission  package  to: 

-  Assess  fixes  to  resolve  RMMV  launch,  handling,  and 
recovery  problems  observed  in  FY14/15  testing  and 
verify  it  can  be  handled  safely  in  higher  sea  states  once 
the  ship’s  structure  is  repaired,  the  weight  restrictions 
are  lifted  and  the  capture  probe  and  latching  mechanism 
(capture  pawls)  are  corrected. 

-  Assess  fixes  to  resolve  communications  problems 
observed  in  FY  13/14/ 15  testing. 

-  Assess  improvements  to  post  mission  analysis  and 
contact  management  software  and  training  to  resolve 
problems  observed  in  1QFY15  testing  when  attempting 
to  pass  RMS  contacts  to  another  mission  system  for 
follow-on  action. 

-  Verify  the  RMS  mission  module  is  ready  to  support 
conduct  of  operational  testing  of  the  LCS  MCM  mission 
package  in  FY15. 
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RQ-21 A  Blackjack 

(formerly  Small  Tactical  Unmanned  Aerial  System  (STUAS)) 


Executive  Summary 

•  The  Navy  started  the  RQ-21  A  Blackjack  (formerly 
Small  Tactical  Unmanned  Aerial  System  (STUAS)) 

IOT&E  in  accordance  with  a  DOT&E-approved  Test  and 
Evaluation  Master  Plan  and  test  plan  in  January  2014. 

The  planned  IOT&E  consisted  of  a  land-based  phase 

of  testing  at  Marine  Corps  Air  Ground  Combat  Center, 
Twentynine  Palms,  California,  and  a  ship-based  phase  of 
testing  aboard  an  LPD-17  class  ship.  Originally  scheduled  for 
completion  in  March  2014,  technical  difficulties,  including 
system  deficiencies,  extended  the  test  into  December  2014. 

•  Concurrent  with  the  land-based  phase  of  IOT&E,  the  Navy 
conducted  RQ-21  A  Blackjack  ship-based  developmental 
testing.  This  ship  testing  identified  interference  between  the 
ships  degaussing  system  and  the  air  vehicle’s  magnetometer. 
This  deficiency  necessitated  software  upgrades  and  regression 
testing,  which  delayed  the  scheduled  ship-based  phase  of 
IOT&E. 

•  The  RQ-21  A  Blackjack  system  provides  commanders 
with  a  long-endurance  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  asset.  The  RQ-21  A  air  vehicle  has  a 
demonstrated  endurance  of  more  than  12  hours  to  provide 
extended  dwell  times  in  support  of  ground  units. 

•  During  the  land-based  phase  of  IOT&E,  the  system 
demonstrated  a  Mean  Flight  Hours  Between  Abort  (MFHBA) 
of  15.8  hours  versus  the  threshold  criterion  of  50  hours.  Low 
reliability  adversely  affected  the  ability  of  operators  to  support 
ground  units  in  a  timely  manner.  In  particular,  air  vehicle 
engines  fail  often,  and  the  system  software  is  immature. 

•  The  Navy  should  conduct  an  in-depth  root  cause  analysis  of 
the  propulsion  module  to  improve  engine  life  so  that  a  greater 
percentage  of  modules  reach  the  advertised  high  time  limits, 
and  should  review  quality  control  procedures  to  reduce  the 
number  of  failed  parts  delivered  to  fielded  systems. 

System 

•  Each  system  consists  of  five  RQ-21  A  unmanned  air  vehicles, 
surface  components,  and  assorted  government-provided 
equipment.  The  surface  components  consist  of  ground  control 
stations,  launch  and  recovery  equipment,  datalinks,  multi 
mission  payloads,  and  support  systems.  Government-provided 
equipment  includes  vehicles  and  generators  to  transport 

and  power  ground  components  as  well  as  intelligence 
workstations. 

•  The  Marine  Corps  intends  the  RQ-21  A  system  to  have  the 
following  capabilities: 

-  Reliability  to  support  an  operating  tempo  of  12  hours  on 
station  per  day  at  a  sustained  rate  for  30  days,  and  the 


capability  for  one  surge  of  24  hours  on  station  per  day  for  a 
10-day  period  during  any  30-day  cycle 

-  Air  vehicle  with  10  hours  endurance,  airspeed  up  to 
80  nautical  miles  per  hour,  and  a  service  ceiling  of 
15,000  feet  density  altitude 

-  Operating  radius  of  50  nautical  miles 

-  Electro-optical  sensor  capable  of  identifying  a  1 -meter 
sized  object  from  3,000  feet  altitude;  infrared  sensor 
capable  of  identifying  a  3 -meter  sized  object  from 
3,000  feet  altitude 

-  Entire  system  transportable  by  CH-53E  helicopter 

•  The  RQ-21  A  Blackjack  will  replace  the  Shadow  RQ-7 
unmanned  aerial  vehicles  (UAVs)  currently  operated  by 
Marine  UAV  Squadrons. 

Mission 

•  Marine  Corps  commanders  will  use  the  RQ-21A  Blackjack 
to  provide  units  ashore  with  a  dedicated  persistent  battlefield 
ISR  capability  that  will  reduce  their  dependence  on  higher 
headquarters  for  ISR  support. 

•  The  persistence  of  the  system  allows  commanders  greater 
coverage  of  their  areas  of  interest,  while  providing  the 
capability  to  concentrate  for  longer  periods  of  time  on  a 
specified  target  of  interest. 

•  In  addition  to  operating  from  land  bases,  detachments  from 
Marine  Corps  UAV  Squadrons  will  embark  the  requisite 
personnel  and  equipment  aboard  L-class  ships  and  conduct 
operations  in  the  maritime  domain. 

Major  Contractor 

Insitu,  Inc.  -  Bingen,  Washington 
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Activity 

•  The  Navy  started  the  RQ-21A  Blackjack  IOT&E  in 
accordance  with  a  DOT&E-approved  Test  and  Evaluation 
Master  Plan  and  test  plan  in  January  2014.  Testing  consisted 
of  a  land-based  IOT&E  phase  (with  concurrent  ship-based 
developmental  testing)  intended  to  be  followed  by  a 
ship-based  IOT&E  phase  aboard  an  LPD-17  class  ship. 

-  During  the  land-based  phase  of  IOT&E  at  Marine  Corps 
Air  Ground  Combat  Center,  Twentynine  Palms,  operators 
flew  188  flight  hours  during  3 1  flights.  The  first  flight 
ended  in  a  mishap  and  loss  of  the  air  vehicle.  Post-mishap 
investigation  suspended  operational  test  flights  for  10  days. 

-  Concurrent  with  the  land-based  phase  of  IOT&E,  the  Navy 
conducted  ship-based  developmental  testing  aboard  an 
LPD-17  class  ship.  This  ship  testing  identified  interference 
between  the  ship’s  degaussing  system  and  the  air  vehicle’s 
magnetometer.  This  deficiency  necessitated  software 
upgrades  and  regression  testing,  which  delayed  the 
scheduled  ship-based  phase  of  IOT&E. 

-  Based  on  poor  system  performance  during  the  land-based 
phase  of  IOT&E  and  a  software  update  to  correct  a  GPS 
deficiency  associated  with  shipboard  operations,  the 
Navy  conducted  a  second  land-based  phase  of  IOT&E  in 
June  at  Marine  Corps  Base  Camp  Lejeune,  North  Carolina. 
Operators  flew  20.9  hours  during  8  flights. 

•  In  May  2014,  the  Marine  Corps  deployed  an  Early  Operational 
Capability  (EOC)  system  (the  equivalent  of  two  RQ-21A 
systems)  to  support  combat  operations  in  Afghanistan. 

The  Marine  Corps  planned  for  this  detachment  to  conduct 
operations  until  the  end  of  September  2014.  The  EOC  systems 
flew  995  hours  in  115  flights  before  termination  3  weeks  early. 

Assessment 

•  The  RQ-21A  Blackjack  provides  the  Marine  Corps  with  a 
cost-effective  alternative  to  the  current  contractor-owned 
and  operated  services  providing  ISR  services  overseas.  A 
Marine  Corps-owned  and  operated  system  provides  for  greater 
flexibility  in  the  deployment  and  location  of  a  commander’s 
organic  ISR  asset. 

•  The  system  provides  the  Marine  Corps  with  a  tactical 
unmanned  aircraft  system  capable  of  launching  and  recovering 
from  aviation  capable  ships.  This  capability  provides  the 
commander  with  an  organic  pre -amphibious  assault  ISR  asset 
that  is  able  to  seamlessly  transition  ashore. 

-  For  tactical  movement  ashore,  the  Marine  Corps  certified 
all  RQ-21A  Blackjack  components  for  transport  by 
CH-53E.  Once  ashore,  a  4  air  vehicle,  10-person 
detachment  is  transportable  by  4  HMMWVs  and  2  trailers. 
While  this  provides  for  a  robust  ISR  capability,  such  a 
detachment  is  dependent  upon  other  units  for  electrical 
power,  shelter,  and  logistical  support  (i.e.,  fuel,  food, 
water). 

-  The  RQ-21 A  Blackjack  is  capable  of  operating  from 
austere  launch  and  recovery  sites.  Because  the  system 
does  not  require  a  runway  for  launch  and  recovery, 


commanders  have  greater  flexibility  in  terms  of  where  to 
base  the  system. 

•  The  modular  nature  of  the  RQ-21  A  air  vehic  le  makes  it 
an  easy  to  maintain  system  at  the  unit  level.  Unit-level 
maintainers  easily  correct  faults  by  removing  and  replacing  the 
affected  module  in  a  minimum  amount  of  time. 

•  During  the  land-based  phase  of  IOT&E,  the  system 
demonstrated  an  MFHBA  of  15.8  hours,  well  below  the 
MFHBA  threshold  criterion  of  50  hours.  Low  reliability 
adversely  affected  the  ability  of  operators  to  support  ground 
units  in  a  timely  manner. 

•  Land-based  IOT&E  identified  production  quality  control 
deficiencies,  such  as  an  incorrectly  assembled  air  vehicle 
fueler,  a  mis-wired  operator  workstation,  and  loose  and  broken 
servo  connectors  in  several  wings.  Some  spare  parts  received 
during  the  test  arrived  with  either  no  software  loaded,  or  the 
wrong  software  version  installed. 

•  Propulsion  Module  Units  (air  vehicle  engines)  are  not  reliable. 
Each  engine  has  a  design  high  time  limit  of  1 00  hours,  which 
means  it  is  designed  to  fly  100  hours  and  then  be  replaced. 
Each  propulsion  unit  accumulates  16  operating  hours  during 
production  and  operational  checks  before  delivery,  resulting  in 
84  hours  of  useful  life  after  delivery.  During  the  land-based 
phase  of  IOT&E,  the  test  unit  used  all  three  spare  engines 
during  the  course  of  188  flight  hours.  Due  to  the  subsequent 
lack  of  spare  engines,  one  air  vehicle  remained  down  for  a 
week  when  its  engine  reached  the  high  time  limit. 

•  The  May  2014  EOC  deployment  also  revealed  problems  with 
propulsion  modules: 

-  Two  propulsion  modules  with  the  deployed  EOC  system 
failed  during  flight,  resulting  in  air  vehicle  loss.  Shortly 
thereafter,  maintainers  discovered  cracks  in  the  generator 
drive  pulley  on  one  propulsion  module  as  they  prepared 
an  air  vehicle  for  flight.  Subsequent  inspections  revealed 
similar  cracks  on  other  propulsion  modules.  As  a  result, 
the  Marine  Corps  reduced  propulsion  module  operating  life 
to  50  hours.  Because  of  propulsion  module  concerns  and 
lack  of  spares,  the  Marine  Corps  terminated  EOC  flight 
operations  three  weeks  early. 

-  The  deployed  EOC  system  replaced  28  propulsion  modules 
during  the  course  of  995  flight  hours.  This  resulted  in  an 
average  useful  life  of  35  hours  per  replaced  propulsion 
module.  The  EOC  unit  replaced  three  of  the  28  propulsion 
modules  because  of  the  100-hour  high  time  limit.  The  unit 
replaced  nine  propulsion  modules  because  of  fuel  leaks. 

•  The  current  communications  relay  payload  does  not  allow 
in-flight  frequency  changes.  Before  flight,  operators  load  a 
single  frequency  into  each  relay  radio.  Any  units  that  require 
the  communications  relay  payload  to  support  their  operations 
must  operate  on  these  frequencies.  This  design  precludes  the 
system  from  supporting  unplanned  communications  relay 
missions  because  supported  units  would  have  to  switch  their 
communications  nets  to  the  two  pre-programmed  payload 
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frequencies.  Tactically,  this  is  something  that  ground  units 
would  not  do. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  made 
progress  in  the  FY13  recommendation  that  the  program 
accelerate  annual  operating  hours  in  order  to  reach  the 
projected  3,300  flight  hours  sooner  than  2017,  which  would 
allow  the  Navy  to  identify  and  correct  failure  modes  before 
committing  to  buy  a  significant  number  of  systems.  In 
May  2014,  the  Marine  Corps  had  deployed  the  equivalent 
of  two  RQ-2 1 A  systems  to  Afghanistan.  This  deployed 
system  has  identified  numerous  failure  modes  not  previously 
encountered.  The  remaining  FY13  recommendation  to 


conduct  a  comprehensive  review  of  RQ-2 1 A  reliability  versus 

requirements  remains  open. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Review  quality  control  procedures  to  reduce  the  number  of 
failed  parts  delivered  to  fielded  systems. 

2.  Conduct  an  in-depth  root  cause  analysis  of  the  propulsion 
module  to  improve  engine  life  so  that  a  greater  percentage 
of  modules  reach  the  advertised  high  time  limits. 

3.  Upgrade  the  communications  relay  payload  to  allow  for 
in-flight  frequency  changes. 

4.  Delay  the  programs  Full-Rate  Production  decision  until 
additional  land-based  operational  testing  verifies  correction 
of  those  deficiencies  identified  during  OT&E. 
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Standard  Missile-6  (SM-6) 


Executive  Summary 

•  The  performance  deficiency  discovered  during  IOT&E  and 
outlined  in  the  classified  Standard  Missile-6  (SM-6)  IOT&E 
report  remains  unresolved  and  continues  to  affect  DOT&E’s 
final  assessment  of  effectiveness. 

-  The  Navy  is  assessing  several  options  for  a  solution,  each 
with  varying  degrees  of  complexity.  A  primary  concern 
is  to  ensure  the  solution  causes  no  degradation  to  the 
existing  SM-6  performance  envelope.  The  Navy  plans  to 
incorporate  these  changes  in  Block  I  (BLK I)  and  Block  IA 
(BLK  IA)  production  variants  in  FY16. 

•  The  Navy  will  not  demonstrate  the  SM-6  Capability 
Production  Document  performance  requirement  for 
interoperability  until  the  fielding  of  the  Navy  Integrated 
Fire  Control  -  Counter  Air  (NIFC-CA)  From-the-Sea  (FTS) 
Increment  I  capability  in  FY15.  The  Navy  expects  to 
demonstrate  the  maximum  range  Key  Performance  Parameters 
during  SM-6  FOT&E  and  Aegis  Baseline  9  operational 
testing  in  FY15,  and  the  launch  availability  Key  Performance 
Parameter  in  the  FY15-16  timeframe. 

•  The  Navy  commenced  developmental  testing  of  pre-planned 
product  improvements  to  the  SM-6  BLK  I  missile;  these 
improvements  are  the  SM-6  BLK  IA  configuration.  A 
successful,  pre-production  developmental  flight  test  (Guidance 
Test  Vehicle-1  (GTV-1))  occurred  in  FY14.  The  Navy  will 
conduct  an  additional  two  GTV  missions,  one  each  in  FY15 
and  FY16,  with  operational  testing  of  the  SM-6  BLK  IA 
planned  for  FY16. 

•  The  Navy  conducted  seven  SM-6  missiles  tests  during  FY14; 
all  missions  were  successful.  Four  of  the  launches  supported 
FOT&E  with  Aegis  Baseline  9,  and  three  supported  NIFC-CA 
FTS  Increment  I  capability. 

•  NIFC-CA  FTS  Increment  I  test  events  have  demonstrated 
a  basic  capability,  but  its  effectiveness  under  operationally 
realistic  conditions  is  undetermined. 

•  DOT&E  continues  to  monitor  the  uplink/downlink  antenna 
shroud  reliability  during  FOT&E.  There  are  no  recorded 
failures  in  testing  since  IOT&E  in  2011. 

System 

•  SM-6  is  the  latest  evolution  of  the  Standard  Missile  family  of 
fleet  air  defense  missiles. 

•  The  SM-6  is  employed  from  cruisers  and  destroyers  equipped 
with  Aegis  combat  systems. 


•  The  SM-6  seeker  and  terminal  guidance  electronics  derive 
from  technology  developed  in  the  Advanced  Medium-Range 
Air-to-Air  Missile  program.  SM-6  retains  the  legacy  Standard 
Missile  semi-active  radar  homing  capability. 

•  SM-6  receives  midcourse  flight  control  from  the  Aegis 
Combat  System  via  ship’s  radar;  terminal  flight  control  is 
autonomous  via  the  missile’s  active  seeker  or  supported  by  the 
Aegis  Combat  System  via  the  ship’s  illuminator. 

•  SM-6  is  being  upgraded  to  the  BLK  IA  configuration  to 
address  hardware  and  software  improvements  and  to  address 
advanced  threats 

Mission 

•  The  Joint  Force  Commander/Strike  Group  Commander  will 
use  SM-6  for  air  defense  against  fixed-/rotary-winged  targets 
and  anti-ship  missiles  operating  at  altitudes  ranging  from  very 
high  to  sea-skimming. 

•  The  Joint  Force  Commander  will  use  SM-6  as  part  of  the 
NIFC-CA  FTS  operational  concept  to  provide  extended  range, 
over-the-horizon  capability  against  at-sea  and  overland  threats. 

Major  Contractor 

Raytheon  Missile  Systems  -  Tucson,  Arizona 


Activity 

•  The  Navy  conducted  seven  SM-6  BLK  I  missiles  tests  during 
FY14  in  accordance  with  a  DOT&E-approved  test  plan;  all 
missions  were  successful.  Four  of  the  launches  supported 


FOT&E  with  Aegis  Baseline  9,  and  three  supported  NIFC-CA 
FTS  Increment  1  capability.  While  the  SM-6  BLK  IA  test 
flight  was  a  developmental  test  not  covered  under  a  DOT&E 
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operational  test  plan,  suitability  data  on  common  components 
were  collected. 

SM-6  BLK I  FOT&E 

•  In  June  2014,  an  SM-6  successfully  engaged  a  high-altitude 
supersonic  target  at  Point  Mugu,  California. 

•  In  August  2014,  an  SM-6  successfully  engaged  a  subsonic 
target  flying  at  the  minimum  altitude  overland  at  White 
Sands  Missile  Range,  New  Mexico. 

•  In  September  2014,  an  SM-6  successfully  engaged  a 
low-altitude  supersonic  target  in  a  crossing  engagement  at 
Point  Mugu,  California. 

•  In  September  2014,  an  SM-6  successfully  engaged  a 
low-altitude  subsonic  target  in  a  crossing  engagement  at 
Point  Mugu,  California. 

NIFC-CA  Increment  I  Flights 

•  In  June  2014,  an  SM-6  BLK  I  successfully  engaged  a 
subsonic  low  radar  cross  section  target  at  medium-range 
and  low-altitude  at  Point  Mugu,  California. 

•  In  June  2014,  an  SM-6  BLK  I  successfully  engaged  a 
subsonic  large  radar  cross  section  target  at  medium-range 
and  medium-altitude  at  Point  Mugu,  California,  . 

•  In  June  2014,  an  SM-6  BLK  I  successfully  engaged  a 
subsonic  large  radar  cross  section  target  at  long-range  and 
medium-altitude  at  Point  Mugu,  California.  This  was  the 
longest-range  engagement  by  SM-6  to-date. 

SM-6  BLK  IA 

•  In  August  2014,  the  Navy  successfully  conducted  a 
land-based  test  launch  of  the  pre-production  SM-6  BLK  IA 
at  White  Sands  Missile  Range,  New  Mexico.  The  missile 
successfully  engaged  a  high-altitude  subsonic  target 
overland.  The  Navy  will  conduct  two  additional  GTV 
missions,  one  each  in  FY15  and  FY16,  the  last  of  which 
will  be  the  production  configuration. 

Assessment 

•  The  FY14  SM-6  BLK  I  flight  tests  were  successful  with  no 
occurrences  of  the  uplink/downlink  antenna  shroud  reliability 
deficiency.  DOT&E  and  the  Navy  will  continue  to  collect 
data  on  this  deficiency  throughout  FOT&E  flight-testing.  In 
addition,  there  were  no  observations  of  additional  anomalies 
during  these  tests. 


•  In  the  May  2013  SM-6  IOT&E  report,  DOT&E  assessed  SM-6 
BLK  I  as  suitable.  This  assessment  considered  combined 
data  from  the  IOT&E  and  developmental/operational  flight 
tests.  During  FY14  testing,  DOT&E  collected  additional 
reliability  data  and  assessed  the  SM-6  BLK  I  continues  to 
remain  suitable.  DOT&E  will  continue  to  collect  suitability 
and  effectiveness  data  throughout  SM-6  BLK  I  FOT&E  testing 
in  FY14/15. 

•  The  performance  deficiency  discovered  during  IOT&E  and 
outlined  in  the  classified  SM-6  BLK  I  IOT&E  report  remains 
unresolved  and  continues  to  affect  DOT&E’s  final  assessment 
of  effectiveness.  The  Navy  is  assessing  several  options  for  a 
solution,  each  with  varying  degrees  of  complexity.  A  primary 
concern  is  to  ensure  the  solution  causes  no  degradation  to  the 
existing  SM-6  performance  envelope.  The  corrective  actions 
will  be  incorporated  into  production  of  the  SM-6  BLK  I  and 
BLK  IA  configurations  and  tested  during  FOT&E  in  FY16. 

•  The  Navy  expects  to  demonstrate  the  maximum  range  Key 
Performance  Parameter  during  SM-6  FOT&E  and  Aegis 
Baseline  9  operational  testing  in  FY15  and  the  launch 
availability  Key  Performance  Parameter  in  the  FY15-16 
timeframe. 

•  The  Navy  will  not  demonstrate  the  SM-6  Capability 
Production  Document  performance  requirement  for 
interoperability  until  the  fielding  of  the  NIFC-CA  FTS 
Increment  I  capability  in  FY15. 

•  The  NIFC-CA  FTS  Increment  I  demonstrations  conducted 
during  FY14  were  basic  developmental  tests  not  conducted 
in  an  operationally  realistic  manner.  The  Navy  plans 

to  continue  testing  the  Increment  I  configuration  with 
increasingly-challenging  scenarios;  however,  no  operational 
test  concept  or  test  plans  for  NIFC-CA  FTS  increments  have 
been  provided  to  DOT&E. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  not 
addressed  the  previous  recommendation  to  complete  corrective 
actions  of  the  classified  performance  deficiency  discovered 
during  IOT&E  and  develop  a  flight  test  program  to  test  those 
corrective  actions. 

•  FY 14  Recommendations.  None. 
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Surface  Electronic  Warfare  Improvement  Program 

(SEWIP)  Block  2 


Executive  Summary 

•  The  Commander,  Operational  Test  and  Evaluation  Force 
(COTF)  conducted  the  IOT&E  of  the  AN/SLQ-32  Electronic 
Warfare  System  (EWS),  equipped  with  the  Surface 
Electronic  Warfare  Improvement  Program  (SEWIP)  Block  2 
upgrade,  from  August  22  through  November  7,  2014,  on 
USS  Bainbridge  (DDG  96)  in  the  Virginia  Capes  operating 
area. 

•  Analysis  of  the  IOT&E  data  is  ongoing.  While  preliminary 
results  indicate  that  the  AN/SLQ-32  EWS  equipped  with  the 
SEWIP  Block  2  upgrade  appears  to  provide  more  capability 
than  the  legacy  AN/SLQ-32  EWS,  the  at-sea  portion  of 

the  IOT&E  on  USS  Bainbridge  from  August  22  -  29 
was  inadequate  to  allow  a  determination  of  operational 
effectiveness. 

•  Inadequate  crew  training  and  proficiency  caused  the  COTF 
test  team  to  provide  operationally  unrealistic  assistance  to 
the  crew,  which  adversely  influenced  some  test  results.  An 
operational  test  of  those  affected  portions  of  the  test  must 
be  conducted  before  operational  effectiveness  can  be  Hilly 
assessed. 

•  In  addition,  several  operational  suitability  deficiencies  have 
been  identified  that  must  be  corrected  and  demonstrated  in  a 
follow-on  operational  test. 

System 

•  SEWIP  is  an  incremental  development  program  that  is 
intended  to  improve  the  electronic  warfare  capability  on  all 
Navy  surface  combatants. 

•  The  SEWIP  Block  1  upgrade  focused  on  the  replacement  of 
obsolete  parts  in  the  AN/SLQ-32  in  addition  to  incorporation 
of  a  new  operator  console,  a  specific  emitter  identification 
capability,  and  an  embedded  trainer. 


•  The  SEWIP  Block  2  upgrade  incorporates  a  new  antenna 
system  and  enhanced  processing  capabilities,  which  are 
intended  to  improve  the  AN/SLQ-32’s  passive  electronic 
support  capabilities. 

•  The  SEWIP  Block  3  upgrade,  which  is  in  early  development, 
will  incorporate  improvements  to  the  AN/SLQ-32 ’s  active 
electronic  attack  to  improve  the  ships’  self-defense  capabilities. 

Mission 

Navy  surface  ships  will  use  SEWIP  to  enhance  their  AN/SLQ-32 
Electronic  Warfare  System  anti-ship  missile  defense, 
counter-targeting,  and  counter-surveillance  capabilities,  as  well 
as  to  improve  the  system’s  ability  to  collect  electronic  data. 

Major  Contractor 

Lockheed-Martin  -  Syracuse,  New  York 


Activity 

•  COTF  conducted  operational  testing  of  the  AN/SLQ-32 
EWS  with  the  SEWIP  Block  2  upgrade  from  August  22 
through  November  7,  2014.  COTF  conducted  the  IOT&E 
onboard  USS  Bainbridge  (DDG  96)  in  the  Virginia  Capes 
operating  area  and  it  included  operationally  representative 
activities  and  scenarios  using  Navy  operators. 

•  A  DOT&E  operational  test  report  is  anticipated  in  2QFY15. 

Assessment 

•  Analysis  of  the  IOT&E  data  is  ongoing.  While  preliminary 
results  indicate  that  the  AN/SLQ-32  EWS  equipped  with  the 
SEWIP  Block  2  upgrade  appears  to  provide  more  capability 


than  the  legacy  AN/SLQ-32  EWS,  the  at-sea  portion  of 
the  IOT&E  on  USS  Bainbridge  from  August  22  -  29 
was  inadequate  to  allow  a  determination  of  operational 
effectiveness. 

•  Inadequate  crew  training  and  proficiency  caused  the  COTF 
test  team  to  provide  operationally  unrealistic  assistance  to 
the  crew,  which  adversely  influenced  some  test  results. 

-  These  included  providing  assistance  in  setting  up 
displays  to  make  threat  detection  easier  to  recognize, 
showing  operators  which  threats  needed  identification, 
helping  operators  identify  extraneous  emitter  contacts, 
calibrating  the  system  when  the  operators  did  not 
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perform  the  calibration  when  the  system  called  for  it,  and 
resetting  the  system  in  the  form  of  unscheduled  warm  and 
cold  starts. 

-  An  operational  test  of  those  affected  portions  of  the  test 
must  be  conducted  before  operational  effectiveness  can  be 
fully  assessed. 

•  Several  operational  suitability  deficiencies  that  included 
several  display  freezes,  system  crashes,  and  unscheduled  warm 
and  cold  starts  have  been  identified  that  must  be  corrected  and 
demonstrated  in  a  follow-on  operational  test. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has 
not  resolved  the  following  FY06  and  FY08  SEWIP 
recommendations : 

1.  Review  and  modify  the  SEWIP  detection  and  classification 
algorithms  to  correct  deficiencies  discovered  while 
operating  in  dense  pulse  and  emitter  environments.  Verify 
the  correction  of  these  deficiencies  during  future  SEWIP 
operational  test  and  evaluation. 


2.  Continue  to  collect  in-service  SEWIP  hardware  reliability 
data  to  gain  a  higher  degree  of  confidence  regarding 
achievement  of  this  requirement. 

3.  Continue  to  review  and  modify  the  SEWIP  software  to 
improve  its  reliability. 

4.  Develop  threat  representative  aerial  target/threat  seeker 
combinations  and/or  procure  actual  threat  anti-ship  cruise 
missiles  for  more  realistic  testing  of  future  SEWIP  block 
upgrades  and  other  electronic  warfare  systems. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Identify  the  portions  of  the  operational  effectiveness  testing 
that  were  adversely  affected  by  the  operationally  unrealistic 
assistance  and  schedule  an  additional  operational  test  of 
those  affected  portions. 

2.  Correct  the  identified  operational  suitability  deficiencies  and 
schedule  a  follow-on  operational  test  period  to  demonstrate 
the  corrections. 
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Surface  Ship  Torpedo  Defense  (SSTD)  System: 
Torpedo  Warning  System  (TWS)  and 
Countermeasure  Anti-Torpedo  (CAT) 


Executive  Summary 

•  The  Navy  fielded  the  Torpedo  Warning  System  (TWS) 
and  the  Countermeasure  Anti-Torpedo  (CAT)  Engineering 
Development  Model  (EDM)-2  with  the  Stored 
Chemical  Energy  Propulsion  System  (SCEPS)  aboard 
USS  George  H.  W.  Bush  when  she  deployed  in  February 
2014  to  Fifth  Fleet  Operating  Areas.  USS  George  H.  W. 

Bush  returned  home  in  November  2014.  Analysis  of  the  data 
collected  during  her  deployment  remains  in  progress. 

•  DOT&E  issued  a  classified  Early  Fielding  Report  on  TWS 
and  CAT  in  April  2014.  DOT&E  determined  the  system 
demonstrated  some  capability  to  detect  certain  types  of  threat 
torpedoes.  Flowever,  the  system  has  not  been  fully  tested  and 
most  TWS  and  CAT  testing  to  date  have  been  conducted  in 
areas  with  benign  acoustic  conditions  when  compared  to  the 
expected  threat  operating  areas,  which  may  have  biased  the 
results. 

•  The  Navy  discovered  an  anomaly  in  the  CAT’s  Safety  and 
Arming  device  in  March  2014,  which  would  significantly 
reduce  the  effectiveness  of  the  CAT.  DOT&E  issued  a 
classified  update  to  the  Early  Fielding  Report  on  TWS  and 
CAT  in  August  2014. 

•  In  November  2014,  the  Navy  conducted  a  new  Quick  Reaction 
Assessment  (QRA)  event  of  a  temporary-installation  version 
of  TWS  and  CAT  (designated  as  a  roll-on/roll-off  system) 
aboard  USS  Theodore  Roosevelt. 

-  The  QRA  event  was  held  in  conjunction  with  a  contractor 
test  event  designated  CT-2.  Only  two  of  the  four  planned 
QRA  events  and  one  of  the  five  planned  CT-2  events 
were  accomplished  due  to  several  factors,  including  poor 
weather.  This  TWS  installation  included  the  new  TWS 
active  sonar. 

-  During  each  completed  event,  a  surrogate  threat  torpedo 
was  fired  at  USS  Theodore  Roosevelt  for  the  TWS  system 
to  detect  and  target.  USS  Theodore  Roosevelt’s  crew, 
with  the  contractor  support  that  will  accompany  the  ship 
on  their  deployment,  engaged  the  threat  torpedo  surrogate 
with  a  CAT  during  some  of  the  events.  All  CATs  that 
were  fired  used  electric  propulsion.  Analysis  of  the  three 
completed  events  is  in  progress. 

•  The  Navy  plans  to  field  the  temporary-installation  version  of 
TWS  and  CAT  installed  aboard  USS  Theodore  Roosevelt  when 
she  deploys  in  FY15. 

System 

•  The  Surface  Ship  Torpedo  Defense  is  a  system-of-systems 
that  includes  two  new  sub-programs:  the  TWS  program  (an 


CAT  -  Countermeasure  Anti-Torpedo  Torpedo 

RSG  •  Ready  Stow  Group 

TAG  •  Target  Acquisition  Group 

TCG  -  Tactical  Control  Group 

TWS  -  Torpedo  Warning  System 

Acquisition  Category  III  program)  and  CAT  (will  not  become 
an  acquisition  program  until  FY16). 

•  TWS  is  being  built  as  an  early  warning  system  to  alert  on  and 
localize  incoming  threat  torpedoes  and  consists  of  three  major 
subsystems: 

-  The  Target  Acquisition  Group  consists  of  a  towed 
acoustic  array,  tow  cable,  winch,  power  supply,  and 
signal-processing  equipment.  Data  from  the  array  and 
the  ship’s  radar  system  are  processed  into  contact  tracks 
and  alerts  to  be  forwarded  to  the  Tactical  Control  Group. 
The  array  will  be  capable  of  both  passive  and  active  sonar 
operations. 

-  The  Tactical  Control  Group  consists  of  duplicate  consoles 
on  the  bridge  and  Combat  Direction  Center  (on  CVNs) 
that  displays  contacts,  issues  torpedo  alerts  to  the  crew, 
and  automatically  develops  CAT  placement  presets  using 
information  sent  from  the  Target  Acquisition  Group. 

The  operator  will  use  this  console  to  manage  the  threat 
engagement  sequence  and  command  CAT  launches. 

-  The  Ready  Stow  Group  will  consist  of  the  steel  cradles 
housing  the  CATs. 

•  CAT  is  a  hard-kill  countermeasure  intended  to  neutralize  threat 
torpedoes  and  consists  of  the  following: 

-  The  Anti-torpedo  Torpedo  (ATT)  is  a  6.75-inch  diameter 
interceptor  designed  for  high-speed  and  maneuverability  to 
support  rapid  engagement  of  the  threat  torpedo. 

-  The  All-Up  Round  Equipment  consists  of  a  nose  sabot, 
ram  plate,  launch  tube,  muzzle  cover,  breech  mechanism, 
and  energetics  to  encapsulate  and  launch  the  ATT. 

-  The  tactical  CAT  is  powered  by  a  SCEPS.  The  battery 
powered  electric  motor  CAT  is  for  test  purposes  only. 
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•  The  Navy  developed  a  temporary  version  of  TWS 
and  CAT  (designated  a  roll-on/roll-off  system)  in 
addition  to  the  permanent-installation  version.  The  first 
temporary-installation  version  is  also  the  first  TWS  system 
to  incorporate  active  sonar  operations.  The  Navy  intends  for 
this  version  to  provide  the  same  function  as  the  permanent 
one. 

-  The  Ready  Stow  Group  is  eliminated  by  mounting  the 
CAT  All-Up  Round  Equipment  directly  to  the  carrier’s 
deck. 

-  The  Tactical  Control  Group  consists  of  two  consoles 
contained  in  a  container  express  box  located  on  the 
carrier’s  hangar  deck. 

-  The  towed  acoustic  array,  tow  cable,  and  winch  are 
permanently  installed  on  the  carrier’s  fantail. 


Mission 

Commanders  of  nuclear-powered  aircraft  carriers  and  Combat 
Logistic  Force  ships  will  use  Surface  Ship  Torpedo  Defense  to 
defend  against  incoming  threat  torpedoes. 

Major  Contractors 

TWS 

•  3Phoenix  -  Wake  Forest,  North  Carolina 

•  In-Depth  Engineering  -  Fairfax,  Virginia 

•  Pacific  Engineering  Inc.  (PEI)  -  Lincoln,  Nebraska 
CAT 

•  Pennsylvania  State  University  Applied  Research 
Laboratory  -  State  College,  Pennsylvania 

•  PEI  -  Lincoln,  Nebraska 


Activity 

•  The  Navy  has  been  working  on  a  hard-kill  torpedo  defense 
system  for  surface  ships  for  over  10  years,  but  accelerated 
the  development  and  fielding  of  TWS  and  CAT  as  a  result 
of  the  March  2010  sinking  of  the  South  Korean  ship,  ROKS 
Cheonan,  and  a  Navy  Fifth  Fleet  Urgent  Operational  Needs 
Statement.  The  Navy  also  decided  to  have  the  systems  protect 
high-value  ships  (aircraft  carriers  and  combat  logistic  ships) 
rather  than  destroyers  as  originally  planned. 

•  In  November  2013,  the  Navy  conducted  a  QRA  aboard 
USS  George  H.  W.  Bush  in  the  Virginia  Capes  Operating 
Areas.  During  each  event,  a  surrogate  threat  torpedo  was  fired 
at  USS  George  H.  W.  Bush  for  the  TWS  system  to  detect  and 
target.  USS  George  H.  W.  Bush’s  crew,  with  the  contractor 
support  that  accompanied  the  ship  on  their  deployment, 
engaged  the  threat  torpedo  surrogate  with  a  CAT.  During  the 
QRA,  two  representative  tactical  CATs  with  SCEPS  were 
fired;  the  remaining  three  CATs  used  electric  propulsion. 

•  The  Navy  fielded  the  TWS  system  and  the  CAT  EDM-2  with 
the  SCEPS  system  aboard  USS  George  H.  W.  Bush  when  she 
deployed  in  February  2014  to  Fifth  Fleet  Operating  Areas. 

USS  George  H.  W.  Bush  returned  home  in  November  2014, 
and  the  analysis  of  the  data  collected  during  her  deployment  is 
in  progress. 

•  DOT&E  issued  a  classified  Early  Fielding  Report  on  TWS  and 
CAT  in  April  2014. 

•  The  Navy  discovered  an  anomaly  in  the  CAT’s  Safety 
and  Arming  device  in  March  2014.  After  being  briefed 
on  the  anomaly,  DOT&E  issued  a  classified  update  to  the 
Early  Fielding  Report  on  TWS  and  CAT  in  August  2014. 

The  Navy  developed  a  correction  for  the  anomaly  in  the 
CAT  Safety  and  Arming  device  but  could  not  install  the 
correction  in  the  fielded  CATs  due  to  safety  concerns  and 
USS  George  H.  W.  Bush  s  operational  schedule. 

•  During  FY14,  the  Navy  and  DOT&E  started  development 
of  an  enterprise  Test  and  Evaluation  Master  Plan  (TEMP) 
for  the  TWS  and  CAT  systems.  The  Navy  made  their  TWS 


Milestone  B  decision  without  a  TEMP  and  is  not  planning  to 
make  the  CAT  system  an  acquisition  program  until  FY16. 

•  In  June  2014,  the  Navy  and  Pennsylvania  State  University 
Applied  Research  Laboratory  conducted  contractor  and 
developmental  testing  of  CAT  at  the  Dabob  Bay,  Washington, 
and  the  Nanoose  Bay,  British  Columbia,  Canada,  acoustic 
tracking  ranges.  The  Dabob  Bay  test  consisted  of  six 
structured  events  to  develop  the  CAT  EDM-2s  ability  to 
intercept  noisy  and  quiet,  straight-running  and  maneuvering 
targets,  to  collect  CAT  self-noise  data,  and  to  collect  data 
where  two  CATs  tracked  a  single  target.  The  Nanoose  Bay 
testing  included  four  structured  events  to  develop  the  CAT 
EDM-2 ’s  ability  to  detect,  track,  and  intercept  surrogate 
threat  torpedoes  in  the  presence  of  a  CVN,  and  one  event  to 
characterize  the  CVN’s  radiated  noise  signature  and  CAT’s 
active  returns  in  the  vicinity  of  the  CVN’s  hull.  Four  Dabob 
Bay  events  used  electrically-propelled  CATs  (ECATs)  and 
two  used  CAT  EDM-2s  with  the  SCEPS.  Three  Nanoose  Bay 
events  used  ECATs  and  two  events  used  CAT  EDM-2s  with 
the  SCEPS. 

•  In  October  2014,  the  Navy  and  3Phoenix  conducted 
contractor  and  developmental  testing  of  TWS ’s  active  source 
at  Lake  Pend  Oreille,  Idaho.  TWS’s  active  source  was 
redesigned  following  the  failure  of  an  earlier  demonstration 
model  in  2011.  This  was  the  first  in-water  test  of  the 
redesigned  active  source  and  included  data  collection  against 
static  and  dynamic  targets  to  support  further  development. 

•  In  November  2014,  the  Navy  conducted  a  new  QRA  event  of  a 
temporary-installation  version  of  TWS  and  CAT  (designated  as 
a  roll-on/roll-off  system)  aboard  USS  Theodore  Roosevelt. 

-  The  QRA  event  was  held  in  conjunction  with  a  contractor 
test  event  designated  CT-2.  Only  two  of  the  four  planned 
QRA  events  and  one  of  the  five  planned  CT-2  events 
were  accomplished  due  to  several  factors,  including  poor 
weather.  This  TWS  installation  included  the  new  TWS 
active  sonar. 
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-  During  each  completed  event,  a  surrogate  threat  torpedo 
was  fired  at  USS  Theodore  Roosevelt  for  the  TWS  system 
to  detect  and  target.  USS  Theodore  Roosevelt’s  crew,  with 
the  contractor  support  that  will  accompany  the  ship  on  their 
deployment,  engaged  the  threat  torpedo  surrogate  with  a 
CAT  during  some  of  the  events.  All  CATs  that  were  fired 
used  electric  propulsion.  Analysis  of  the  three  completed 
events  is  in  progress. 

•  The  Navy  plans  to  field  the  temporary-installation  version  of 
TWS  and  CAT  installed  aboard  USS  Theodore  Roosevelt  when 
she  deploys  in  FY15. 

Assessment 

•  The  prototype  TWS  and  early  engineering  developmental 
model  CAT  installed  on  USS  George  H.  W.  Bush  and 
USS  Theodore  Roosevelt  demonstrated  some  capability  to 
detect  certain  types  of  threats.  However,  the  system  has  not 
been  fully  tested  and  the  Navy  conducted  most  TWS  and 
CAT  testing  to  date  in  areas  with  benign  acoustic  conditions 
when  compared  to  the  expected  threat  operating  areas,  which 
may  have  biased  the  results  high.  Additionally,  most  threat 
surrogates  were  not  executing  operationally  realistic  threat 
torpedo  profiles  due  to  safety  constraints. 

•  The  Navy’s  decision  to  add  a  highly-trained  contractor  as 
the  acoustic  operator  to  supplement  the  automated  detection 
and  alerting  functions  of  TWS  improved  threat  detection 
performance  during  the  2013  QRA.  However,  the  test  areas 
did  not  offer  the  same  number  of  opportunities  for  false  alerts 
as  expected  in  the  threat  area;  thus,  it  is  not  known  if  the 
presence  of  the  operator  could  also  reduce  the  false  alert  rate. 
For  safety  reasons,  the  QRA  testing  was  highly  structured 
and  allowed  the  operators  to  focus  on  torpedo  detections 
and  firing  the  CAT.  Therefore,  QRA  testing  was  inadequate 
to  resolve  the  rate  of  false  alerts  or  their  impact  on  mission 
accomplishment. 

•  USS  George  H.  W.  Bush’s  deployment  was  useful  in 
identifying  TWS  false  alert  sources,  but  system  development 
done  using  these  data  needs  to  be  assessed  in  testing  that 
includes  the  presence  of  both  threat  torpedo  surrogates  and 
assets  that  may  cause  false  alerts  simultaneously. 

•  During  developmental  testing  and  the  2013  QRAs,  a  properly 
targeted  CAT  EDM-2  demonstrated  a  capability  to  detect 
and  home  on  some  surrogates  torpedoes.  However,  all  of  the 
surrogate  threat  torpedoes  and  CATs  were  operating  deeper 
than  most  expected  threat  torpedoes  due  to  safety  reasons. 
Shallower  scenarios  that  force  the  CAT  to  track  and  attack  the 
surrogate  threat  torpedo  in  the  challenging  areas  of  the  water 
column  were  only  investigated  during  limited  contractor  test 
events  at  Dabob  Bay  and  during  a  single  event  in  Nanoose  Bay 


in  June  2014;  the  Navy  has  not  collected  adequate  data  to 
assess  CAT’s  overall  ability  to  neutralize  these  threats. 

•  The  Navy  developed  a  correction  for  the  anomaly  in  the  CAT 
Safety  and  Arming  device,  but  has  not  yet  implemented  a  way 
to  verify  the  device’s  correct  operation  in  sea  tests. 

•  Completed  testing  indicates  the  new  active  source  has  both 
hardware  and  software  reliability  deficiencies,  which  the  Navy 
is  investigating.  The  temporary-installation  system  exhibited 
other  reliability  deficiencies  with  interfaces  to  ship’s  power, 
operator  display  consoles,  and  the  array-handling  equipment. 
Should  the  Navy  field  the  temporary-installation  prototype 
TWS  and  EDM-2  model  of  the  CAT  aboard  USS  Theodore 
Roosevelt  in  FY15,  this  will  be  the  first  fielding  of  a  TWS  that 
incorporates  active  sonar  operations.  Additional  information 
on  the  testing  of  TWS  and  CAT  performance  will  be  included 
in  DOT&E’s  classified  Early  Fielding  Report  in  2QFY15. 

•  The  ATT  warhead  tests  indicate  that  the  ATT  should  be 
lethal  against  select  representative  torpedo  threats  provided 
that  both  the  CAT’s  closest  point  of  approach  to  the  threat 
torpedo  and  the  CAT’s  fuzing  occurs  within  the  explosive  kill 
zone.  Further  test  and  analysis  is  required  to  determine  the 
comprehensive  lethal  capability  of  the  ATT. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  has  made 
limited  progress  on  the  FY13  recommendations.  The  Navy 
should  still: 

1.  Develop  TEMPs  for  the  TWS  and  CAT  system  and  an 
LFT&E  strategy  for  the  ATT  lethality  as  soon  as  possible. 

2.  Conduct  additional  testing  in  challenging, 
threat-representative  environments. 

3.  Conduct  additional  CAT  testing  using  operationally 
realistic  threat  target  profiles  closer  to  the  surface  to  assess 
the  CAT’s  terminal  homing,  attack,  and  fuzing  within  the 
lethality  range  of  the  warhead. 

•  FY 14  Recommendations.  The  Navy  should: 

1 .  Implement  a  way  to  verify  the  correct  operation  of  the 
CAT’s  Safety  and  Arming  device  in  all  future  sea  tests. 

2.  Investigate  test  methods  designed  to  reduce  or  eliminate 
the  safety  limitations  that  have  previously  prevented 
testing  against  operationally  realistic  target  scenarios.  The 
Navy  should  consider  using  geographic  separation,  range 
boundaries,  and  shallower  draft  ships  for  future  TWS  and 
CAT  testing. 

3.  Investigate,  correct,  and  retest  deficiencies  identified 
with  the  active  source  and  other  components  of  the 
temporary-installation  system  before  fielding  these  aboard 
USS  Theodore  Roosevelt. 
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Surveillance  Towed  Array  Sensor  System  (SURTASS) 
and  Compact  Low  Frequency  Active  (CLFA)  Sonar 


Executive  Summary 

•  DOT&E  issued  a  classified  Early  Fielding  Report  on 
Surveillance  Towed  Array  Sensor  System  (SURTASS) 
and  Compact  Low  Frequency  Active  (CLFA)  Sonar  on 
February  24,  2014.  This  report  was  submitted  due  to 
significant  and  uncertain  delays  in  the  completion  of  IOT&E. 

It  concluded  that  SURTASS/CLFA  was  not  operationally 
effective  during  the  limited  testing  conducted  in  FY12. 

•  The  Commander,  Operational  Test  and  Evaluation  Force 
(COTF)  continued  IOT&E  in  September  2014  in  conjunction 
with  the  fleet  exercise  Valiant  Shield  14  and  a  follow-on 
dedicated  test  phase.  Completion  of  IOT&E  is  expected  in 
FY15. 

System 

•  SURTASS/CLFA  is  a  low-frequency,  passive  and  active, 
acoustic  surveillance  system  installed  on  tactical  auxiliary 
general  ocean  surveillance  (T-AGOS)  ships  as  a  component  of 
the  Integrated  Undersea  Surveillance  System. 

•  SURTASS  provides  passive  detection  of  nuclear  and  diesel 
submarines  and  enables  real-time  reporting  of  surveillance 
information  to  Anti-Submarine  Warfare  (ASW)  commanders. 

•  CLFA  is  a  low-frequency,  active  sonar  system  developed  to 
provide  an  active  detection  capability  of  quiet  submarines 
operating  in  environments  that  support  active  sonar 
propagation. 

•  The  system  consists  of: 

-  A  T-AGOS  host  ship  with  array-handling  equipment 

-  A  towed  vertical  string  of  active  acoustic  projectors 
(CLFA) 

-  A  towed  horizontal  twin  line  (TL-29A)  passive  sonar  array 

-  An  Integrated  Common  Processor  (ICP)  for  processing 
active  and  passive  acoustic  data 

-  A  High-Frequency  Marine-Mammal  Monitoring  (HFM3) 
active  sonar  used  to  ensure  local  water  space  is  free 

of  marine  mammals  prior  to  low-frequency  active 
transmission 

-  A  communications  segment  to  provide  connectivity 
to  shore -based  Integrated  Undersea  Surveillance 
System-processing  facilities  and  to  fleet  ASW  commanders 

Mission 

Maritime  Component  Commanders: 

•  Employ  T-AGOS  ships  equipped  with  SURTASS/CLFA 
systems  to  provide  long-range  active  and  passive  ASW 
detection,  classification,  and  tracking  of  submarines  in  support 
of  Carrier  Strike  Group  and  theater  ASW  operations 


•  Use  SURTASS/CLFA  to  provide  blue  force  ASW  screening 
and  threat  submarine  localization  information  to  theater  ASW 
commanders  to  support  coordinated  prosecution  of  detected 
threat  submarines 

Major  Contractors 

•  Overall  Integrator:  Maritime  Surveillance  Systems  Program 
Office  (PMS  485) 

•  Integrated  Common  Processor:  Lockheed 
Martin  -  Manassas,  Virginia 

•  CLFA  Projectors:  BAE  -  Nashua,  New  Hampshire 

•  CLFA  Handling  System:  Naval  Facilities  Engineering 
and  Expeditionary  Warfare  Center  (NAVFAC  EXWC) 
(Government  Lab)  -  Port  Hueneme,  California 

•  HFM3:  Scientific  Solutions  Incorporated 
(SSI)  -  Nashua,  New  Hampshire 

•  TL-29A  Towed  Arrays:  Lockheed 
Martin  -  Syracuse,  New  York 
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Activity 

•  On  February  24,  2014,  DOT&E  issued  a  classified  Early 
Fielding  Report  for  SURTASS/CLFA.  This  report  was  issued 
due  to  significant  and  uncertain  delays  in  the  completion  of 
IOT&E  that  commenced  in  FY12. 

•  In  September  2014,  COTF  resumed  IOT&E  and  collected 
operational  data  during  two  events.  Testing  was  conducted  in 
accordance  with  a  DOT&E-approved  test  plan. 

-  The  Navy  conducted  four  days  of  ASW  area  search 
operations  for  SURTASS/CLFA  in  support  of  coordinated 
theater  ASW  that  included  surface  and  air  assets  during 
Valiant  Shield  14. 

-  COTF  planned  two,  dedicated  five-day  test  phases  to 
achieve  sufficient  data  to  characterize  SURTASS/CLFA 
detection  capability  against  long-range  submarine 
approaches.  One  of  the  test  phases  was  cancelled  due  to 
the  unavailability  of  the  intended  target  submarine.  During 
the  remaining  test  phase,  COTF  completed  four  events 
during  two  and  a  half  days  with  the  SURTASS/CLFA 
conducting  ASW  large  area  search. 

•  One  engineering  development  model  and  two  production 
CLFA  systems  were  available  for  operation  on  three  of  the  five 
Western  Pacific -based  T-AGOS  ships  during  FY14. 

Assessment 

•  The  final  assessment  of  SURTASS/CLFA  is  not  complete, 
as  testing  is  expected  to  continue  into  FY15.  However,  the 
DOT&E  classified  Early  Fielding  Report  concluded  the 
following  regarding  performance: 

-  SURTASS/CLFA  was  not  operationally  effective 

in  supporting  submarine  prosecution  during  Valiant 
Shield  12.  The  fleet  did  not  demonstrate  the  ability  to 
correlate  non-submarine  CLFA  detections  to  real-time 
surface  ship  positions  during  Valiant  Shield  12.  Failure 


to  exclude  the  numerous  surface  ship  detections,  coupled 
with  limited  AS  W-capable  assets,  will  not  support  fleet 
prosecution  of  CLFA  submarine  localizations.  Further 
details  of  the  observed  deficiencies  are  available  in  the 
classified  report. 

-  Limited  operational  test  data  demonstrated  that  the 
SURTASS/CLFA  is  capable  of  detecting  submarines  at 
long  ranges  using  both  active  and  passive  sonar.  However, 
data  collected  were  insufficient  to  fully  characterize  the 
detection  capability. 

-  SURTASS/CLFA  demonstrated  that  it  can  be  operationally 
suitable.  Poor  reliability  of  HFM3  active  sonar  during 
the  operational  test  significantly  reduced  the  availability 
of  CLFA  and  contributed  to  insufficient  data  collection. 
Federal  law  requires  HFM3  active  sonar  to  mitigate  the 
taking  of  marine  mammals  by  low-frequency  active  sonar, 
but  the  operation  of  this  peacetime  system  does  not  affect 
the  wartime  capability  of  CLFA. 

•  The  analysis  of  operational  test  data  collected  in  FY14  is 
ongoing.  DOT&E  will  report  the  results  in  FY15  at  the 
completion  of  testing. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Navy  should 
continue  to  address  the  remaining  FY13  recommendation 
to  improve  procedures  and  training  for  correlating  CLFA 
non-submarine,  active  detections  with  real-time  surface  vessel 
positions. 

•  FY 14  Recommendation. 

1.  The  Navy  should  address  the  10  classified 

recommendations  listed  in  the  February  2014  Early 
Fielding  Report. 
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AC-130J  Ghostrider 


Executive  Summary 

•  U.S.  Special  Operations  Command  (USSOCOM)  is 
developing  AC-  130J  through  the  integration  of  a  modular 
Precision  Strike  Package  (PSP)  onto  existing  MC-130J 
aircraft.  The  PSP  was  previously  developed  and  tested  on 
several  AC- 130 W  aircraft  since  2009. 

•  The  AC-130J  first  flight  was  in  January  2014. 

•  Developmental  testing  and  evaluation  (DT&E)  identified 
several  problems  that  require  resolution  and  have  delayed  the 
operational  assessment  supporting  the  Milestone  C  decision  by 
approximately  four  months. 

•  The  U.S.  Air  Force  Combat  Effectiveness  and  Vulnerability 
Analysis  Branch  completed  an  initial  qualitative  survivability 
study  of  legacy  aircraft  to  support  the  detailed  AC-130J 
survivability  analysis  and  evaluation  plan  development  as  laid 
out  in  the  Live  Fire  Alternative  Test  Plan. 

System 

•  The  AC-130J  is  a  medium-sized,  multi-engine,  tactical  aircraft 
with  a  variety  of  sensors  and  weapons  for  air-to-ground  attack. 

•  USSOCOM  is  developing  AC-130J  through  the  integration  of 
a  modular  PSP  onto  existing  MC-130J  aircraft.  USSOCOM 
continues  to  develop  new  PSP  capabilities  in  parallel  on 
legacy  AC-130W  aircraft  prior  to  introduction  on  the  AC-130J. 

-  The  current  PSP  provides  a  weapons  suite  composed  of  a 
30  mm  side-firing  chain  gun;  wing-mounted,  GPS-guided 
Small  Diameter  Bombs;  and  Griffin  laser-guided  missiles 
mounted  internally  and  launched  through  the  rear  cargo 
door. 

-  The  PSP  also  provides  an  Intelligence,  Surveillance, 
and  Reconnaissance  suite  composed  of  two 
electro-opticaE infrared  sensor/laser  designator  pods;  a 
synthetic  aperture  radar  pod;  and  multiple  video,  data,  and 
communication  links.  All  PSP  subsystems  are  controlled 
from  a  dual-console  Mission  Operator  Pallet  (MOP)  in 
the  cargo  bay,  with  remote  displays  and  control  panels 
(including  master  arm  and  consent  switches  and  a  gun 
trigger)  on  the  flight  deck. 

-  The  program  intends  to  add  a  105  mm  gun  beginning 
with  aircraft  #3  (scheduled  to  complete  modification 
in  mid-FY16).  Partially,  as  a  result  of  this,  the  crew 
complement  will  increase  from  seven  to  nine,  and  some 
crew  responsibilities  will  change. 

-  Future  updates  will  add  a  laser-guided  variant  of  the 
Small  Diameter  Bomb,  and  wing-mounted  laser-guided 
F1ELLFIRE  missiles. 


•  The  AC-130J  will  retain  the  ability  to  be  refueled  in  flight,  but 
it  will  not  retain  the  external  hose-and-drogue  pods  used  to 
refuel  other  aircraft. 

•  The  AC-130J  retains  all  survivability  enhancement  features 
found  on  the  HC/MC-130J  aircraft.  Susceptibility  reduction 
features  include  the  AN/ALR-56M  radar  warning  receiver, 
AN/AAR-47(V)2  missile  warning  system,  and  AN/ ALE-47 
countermeasures  dispensing  system.  Vulnerability  reduction 
features  include  fuel  system  protection  (fuel  tank  foam  to 
protect  from  ullage  explosion),  redundant  flight-critical 
components,  and  armor  for  crew  and  oxygen  supply 
protection. 

•  The  AC-130J  will  replace  legacy  AC- 13 0H/U  aircraft. 

Mission 

The  Joint  Task  Force  or  Combatant  Commander  will  use: 

•  The  AC-130J  to  provide  persistent  and  precision  strike 
operations,  including  close  air  support,  air  interdiction,  and 
armed  reconnaissance. 

•  The  AC-130J  sensor,  data,  and  communications  suite  to 
provide  battlespace  wide  area  surveillance  and  situational 
awareness;  execute  non-traditional  Intelligence,  Surveillance, 
and  Reconnaissance  operations;  and  support  combat  search 
and  rescue  operations. 

Major  Contractor 

Lockheed  Martin  -  Bethesda,  Maryland 
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Activity 

•  DT&E  began  with  ground  tests  on  the  first  aircraft  in 
October  2013  while  integration  was  being  completed;  flight 
tests  began  in  January  2014. 

•  The  Air  Force  USSOCOM  delivered  the  second  MC-130J  for 
conversion  to  an  AC-130J  in  September  2014. 

•  The  program  selected  a  different  intercommunication  system 
for  aircraft  #2.  This  change,  as  well  as  delayed  delivery  of  the 
necessary  government-furnished  information  and  equipment 
to  the  integrating  contractor,  has  caused  a  projected  two-  to 
three-month  delay  in  availability  of  aircraft  #2  for  testing. 

•  The  LFT&E  Integrated  Program  Team  has  drafted  the 
Ballistic  Vulnerability  Analysis,  the  Anti-Aircraft  Artillery 
Susceptibility  Analysis,  and  the  Proximity  Burst  Analysis 
plans.  The  team  reviewed  and  summarized  the  available 
legacy  aircraft  survivability  data  to  provide  a  more  informed 
and  more  efficient  set  of  analysis  plans  specifically  tailored  to 
the  AC-130J  Concept  of  Operations.  The  execution  of  these 
plans  is  expected  to  begin  in  1QFY15. 

Assessment 

•  DT&E  identified  several  problems  that  require  resolution  and 
will  affect  the  subsequent  development  test  schedule: 

-  In  February  2014,  during  flying  and  handling  qualities 
testing  near  the  stall  limit,  the  aircraft  experienced  a 
temporary  departure  from  controlled  flight.  The  recovery 
maneuver  exceeded  some  speed  and  load  limits  on 

the  aircraft.  Flight  testing  was  suspended  for  aircraft 
inspections  and  a  safety  incident  investigation.  Upon 
return  to  flight,  testing  was  added  to  the  DT&E  plan  to 
characterize  the  flight  envelope  more  carefully.  This  will 
delay  completion  of  DT&E  by  approximately  two  months. 

-  Problems  integrating  the  PSP  weapon  kit  onto  the  aircraft 
continue  to  delay  portions  of  developmental  testing  by 
prohibiting  weapons  employment. 

■  The  visual  acuity  of  the  electro-optical/infrared  sensors 
installed  on  the  AC-130J  is  not  sufficient  for  accurate 
target  identification  and  designation  because  the  new 
aircraft  causes  more  vibration  than  the  legacy  AC-130W 
aircraft  on  which  the  PSP  was  previously  installed. 

■  ElectricaEradio-frequency  interference  between  aircraft 
systems  and  the  hand  controllers  used  by  crewmembers 
to  direct  the  sensors  and  weapons  has  caused  erratic 
sensor  movements.  This  inhibits  target  tracking  and 

is  a  safety  hazard  (risk  of  fratricide)  during  weapon 
employment. 

■  The  program  is  working  on  correcting  the  sensor 
vibration  issue  by  collecting  flight  test  data  that  can  be 
used  by  the  subsystem  contractor  to  develop  mechanical 
and  software  updates  to  reduce  the  effect  of  vibration. 
Similar  efforts  are  underway  to  characterize  and  correct 
electrical  interference  with  the  controllers.  The  program 


has  reported  some  progress  in  the  laboratory  environment 
on  both  issues,  but  definitive  solutions  have  not  yet  been 
demonstrated  on  the  aircraft. 

-  The  program  has  accomplished  36  test  flights  out  of 
approximately  130  flights  planned  for  a  total  of  97  flight 
hours  (including  flights  added  after  the  stall  incident). 

Initial  DT&E  is  now  expected  to  be  completed  in 

May  2015. 

•  Delays  in  developmental  testing  have  delayed  the  planned 
operational  assessment  by  the  1 8th  Flight  Test  Squadron 
by  approximately  four  months.  This  diminishes  the  risk 
mitigation  value  of  the  operational  assessment  to  both  IOT&E 
and  the  Milestone  C  decision.  IOT&E  has  been  delayed  until 
October  2015. 

•  Aircraft  #2  is  planned  to  undergo  a  second  phase  of  DT&E 
prior  to  IOT&E  in  order  to  verify  functionality  of  several 
design  changes  relative  to  aircraft  #1,  ranging  from  minor 
adjustments  made  to  software  and  hardware  integration 
learned  from  the  first  integration  effort  to  more  significant 
changes  like  the  selection  of  a  new  intercommunication 
system.  Since  the  second  aircraft  will  not  be  ready  until 
June  2015,  this  leaves  minimal  time  for  correction  of  any  new 
deficiencies  prior  to  IOT&E  and  very  little  time  for  the  IOT&E 
crew  to  train  on  the  test  aircraft.  This  has  been  partially 
mitigated  by  providing  both  aircraft  #1  and  #2  to  support 
training  for  IOT&E  crews. 

•  The  changes  in  aircraft  configuration  and  crew  complement 
between  aircraft  #2  (the  test  article  for  IOT&E)  and  aircraft  #3 
(the  intended  configuration  for  fielding)  decrease  the 
production-representative  character  of  the  IOT&E  test  article, 
diminish  the  utility  of  any  crew  workload  studies  in  IOT&E, 
and  increase  the  scope  of  the  necessary  FOT&E. 

-  If  IOT&E  cannot  be  deferred  until  aircraft  #3  is  available, 
it  may  be  made  more  operationally  relevant  by  employing 
an  eight-person  crew  that  accounts  for  the  movement 

of  a  combat  systems  operator  to  the  flight  deck  and  the 
addition  of  a  special  mission  aviator  (SMA)  to  the  MOP; 
the  ninth  crewmember  (the  second  added  SMA)  would 
not  be  needed  without  the  105  mm  gun.  This  would  allow 
IOT&E  to  more  accurately  evaluate  tactics  and  workloads 
on  the  flight  deck  and  at  the  MOP,  and  it  would  reduce 
the  scope  of  FOT&E  to  focus  on  the  addition  of  the 
105  mm  gun  and  final  SMA. 

•  Draft  survivability  analysis  plans  are  comprehensive,  informed 
by  legacy  aircraft  survivability  data,  and  consider  a  range  of 
engagement  scenarios.  Most  of  these  analyses  will  be  used  to 
assess  aspects  of  the  AC-130J  survivability  using  its  current 
increment  of  capabilities  but  some  are  also  applicable  to 
future  increment  of  capabilities  (e.g.  the  Commando  II  Radio 
Frequency  Countermeasures  system),  which  is  expected  to  be 
fielded  in  FY19. 
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Recommendations 

•  Status  of  Previous  Recommendations.  The  program  has 
addressed  three  of  the  four  FY13  recommendations,  and  has 
made  progress  on  the  remaining  recommendation  to  collect 
and  provide  DOT&E  with  all  reliability  data  on  the  AC-130W 
that  can  augment  the  suitability  evaluation  for  AC-130J.  The 
Program  Office  provided  some  data  to  DOT&E,  but  the  data 
are  not  complete.  The  Program  Office  will  provide  additional 
data  to  DOT&E  in  time  for  comparison  to  AC-130J  IOT&E 
data. 


FY14  Recommendations.  The  Program  Office  should: 

1.  Schedule  sufficient  time  for  IOT&E  crew  training  prior  to 
the  start  of  IOT&E. 

2.  Consider  entering  IOT&E  with  an  eight-person  crew  to 
more  accurately  evaluate  tactics  and  workloads. 
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AIM-120  Advanced  Medium-Range  Air-to-Air  Missile 

(AMRAAM) 


Executive  Summary 

•  The  Air  Force  and  Navy  completed  FOT&E  of  the  AIM-120 
Advanced  Medium-Range  Air-to-Air  Missile  (AMRAAM) 
in  July  2014  with  the  intent  of  making  a  production  go-ahead 
decision  in  December  2014. 

•  The  AMRAAM  Program  Office  initiated  System  Improvement 
Program  (SIP)-l  testing  in  September  2014.  SIP-1  is  one  of 
several  follow-on  programs  designed  to  enhance  AIM- 120D 
performance. 

•  The  Air  Force  continued  integrated  testing  on  AIM-120 
AMRAAM  Electronic  Protection  Improvement  Program 
(EPIP),  a  software  upgrade  to  AIM-120C3-C7  variants,  under 
a  separate  EPIP  Test  and  Evaluation  Master  Plan  that  DOT&E 
approved  in  April  2012. 

System 

•  AMRAAM  is  a  radar-guided,  air-to-air  missile  with  capability 
in  both  the  beyond  visual-range  and  within  visual-range 
arenas.  A  single-launch  aircraft  can  engage  multiple  targets 
with  multiple  missiles  simultaneously  when  using  AMRAAM. 

•  The  latest  fielded  version,  the  AIM-120C7,  incorporated 
an  upgraded  antenna,  receiver,  signal  processor,  and  new 
software  algorithms  to  counter  new  threats.  The  use  of  smaller 
system  components  created  room  for  future  growth. 

•  The  AMRAAM  program  periodically  develops  and 
incorporates  phased  upgrades.  The  AMRAAM  EPIP  is  a 
software  upgrade  to  AIM-120C3-C7. 

•  The  AIM-120D  is  currently  in  development  and  the  Air  Force 
and  Navy  intend  for  it  to  deliver  performance  improvements 
beyond  the  AIM-120C7  using  an  internal  GPS,  enhanced 
datalink,  improved  kinematics,  and  new  software.  Following 


FOT&E,  the  contractor  will  execute  a  series  of  SIPs  that  will 
consist  of  software  upgrades  to  AIM-120D. 

Mission 

•  The  Air  Force  and  Navy,  as  well  as  several  foreign  military 
forces,  use  various  versions  of  the  AIM- 120  AMRAAM  to 
shoot  down  enemy  aircraft. 

•  All  U.S.  fighter  aircraft  use  the  AMRAAM  as  the  primary, 
beyond  visual-range  air-to-air  weapon. 

Major  Contractors 

•  Raytheon  Missile  Systems  -  Tucson,  Arizona 

•  Rocket  Motor  Subcontractors: 

-  Alliant  Techsystems  (ATK)  -  Arlington,  Virginia 

-  Nammo  (Nordic  Ammunition  Group)  -  Raufoss,  Norway 


Activity 

•  The  Air  Force  and  Navy  conducted  all  testing  in  accordance 

with  the  DOT&E-approved  Test  and  Evaluation  Master  Plan. 

AIM-120D 

•  AIM-120D  operational  testing  consisted  of  multiple  live 
missile  shots  and  captive-carry  events.  The  Air  Force  and 
Navy  completed  AIM- 120D  FOT&E  in  July  2014. 

•  The  Program  Office  plans  to  conduct  SIP- 1  integrated 
testing  with  two  live  missile  shots  during  2QFY15. 
Operational  testing  for  SIP- 1  is  scheduled  to  begin  in 
3QFY15. 

•  The  Air  Force  plans  to  make  a  production  go-ahead 
decision  on  AIM-120D  by  December  2014. 


•  As  of  October  2,  2014,  Raytheon  had  delivered  a  total  of 
1,031  AIM-120Ds  for  the  Air  Force  and  Navy. 

AMRAAM  EPIP 

•  In  October  2014,  the  Air  Force  completed  EPIP  Basic 
Phase  III  operational  testing  for  AIM-120C-7  missiles. 

•  The  Air  Force  and  Navy  began  EPIP  Basic  Phase  II 
operational  testing  for  AIM-120C-3,  -4,  -5,  and  -6  missiles 
in  October  2014. 

Lot  Acceptance  Test/Rocket  Motors 

•  The  Weapon  System  Explosives  Safety  Review  Board 
completed  acceptance  and  concurrence  for  fleet  release 
for  the  ATK  baseline  rocket  motor  in  June  2014.  Alliant 
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Techsystems  is  delivering  rocket  motors  to  the  Navy  and 
Foreign  Military  Sales  customers.  Nammo  is  delivering 
rocket  motors  for  U.S.  AIM-120D  and  AIM-120C-7 
Foreign  Military  Sales  customers. 

Assessment 

•  AMRAAM  continues  to  be  operationally  effective  and 
suitable. 

•  AIM-120D  captive-carry  performance,  as  measured 
during  FOT&E,  exceeded  the  Mean  Time  Between  Failure 
requirement  with  452.5  hours,  exceeding  the  requirement 
of  450  hours  desired  two  years  after  Initial  Operational 
Capability. 

•  The  Air  Force  identified  deficiencies  in  missile  performance 
that  did  not  significantly  degrade  overall  effectiveness.  The 


Air  Force  and  Raytheon  Missile  Systems  developed  solutions 
for  specific  deficiencies  and  will  assess  them  during  SIP-1 
testing. 

•  The  shortage  in  rocket  motors  due  to  unacceptable  Lot 
Acceptance  Test  performance  did  not  significantly  affect 
AIM-120D  testing,  despite  creating  a  backlog  in  production. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 
satisfactorily  addressed  the  previous  recommendations. 

•  FY 14  Recommendation. 

1.  The  Air  Force  should  complete  SIP-1  operational  testing  to 
achieve  the  Service  member’s  desired  mission  effectiveness 
improvements  for  AIM-120D. 
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Air  Force  Distributed  Common  Ground  System 

(AF  DCGS) 


Executive  Summary 

•  The  Air  Force  605th  Test  and  Evaluation  Squadron  (TES) 
conducted  two  phases  of  a  Force  Development  Evaluation 
(FDE)  of  the  latest  version  of  Air  Force  Distributed 
Common  Ground  System  (AF  DCGS)  (Bulk  Release  10B) 
in  January  and  June  2014.  Bulk  Release  10B  was  delivered 
as  a  part  of  a  Geospatial  Intelligence  (GEOINT)  update. 

Key  software  enhancements  of  Bulk  Release  10B  degraded 
system  performance  to  unacceptable  levels,  leading  the  Air 
Force  to  turn  off  the  software  in  order  to  continue  mission 
operations. 

•  The  Air  Force  split  the  AF  DCGS  program  into  four 
Acquisition  Category  (ACAT)  III  programs  after  declaring 
Full  Operational  Capability  in  2009  and  moved  the  program 
from  the  development  phase  into  the  sustainment  phase 
despite  the  program’s  immaturity.  The  four  ACAT  III 
programs  lack  appropriate  and  up-to-date  test  and  evaluation, 
systems  engineering,  and  requirements  documentation. 

System 

•  AF  DCGS  is  an  intelligence  enterprise  system  that  includes 
hardware  housed  in  5  core  sites  and  16  distributed  sites, 
with  a  network  that  connects  them  to  each  other  and  to 
other  intelligence  networks,  sensors,  and  mission  command 
systems. 

•  The  Air  Force  declared  AF  DCGS  to  be  at  Full  Operational 
Capability  in  2009.  Since  then,  the  Air  Force  has  treated 
AF  DCGS  as  if  it  is  in  sustainment,  even  though  it  is 
continuing  to  develop  four  ACAT  III  programs:  Signal 
Intelligence  (SIGINT)  upgrades,  GEOINT  upgrades, 
Network  Communications,  and  Data  Links. 

•  The  GEOINT  upgrade.  Bulk  Release  10B,  introduced 
a  common  baseline  across  the  AF  DCGS  enterprise.  A 
common  baseline  should  allow  for  easier  updates  and 
modifications  of  the  entire  enterprise,  and  provide 
capabilities  to  handle  new  and  emerging  sensor  information 
and  the  increased  data  loads  resulting  from  the  growing 
number  of  sensors  being  fielded.  After  the  Bulk  Release  10B 
FDE  at  DCGS  Ground  station  (DGS)-2,  the  Air  Force 
renamed  the  Bulk  Release  10B  to  GEOINT  Baseline  4.1. 

•  The  AF  DCGS  System  Release  3.0  is  a  SIGINT  upgrade, 
which  makes  SIGINT  data  and  services  available  to  internal 
and  external  users;  improves  operations  with  the  Airborne 
Signals  Intelligence  Payload  low-band  sensor;  and  improves 
processing,  exploitation,  and  dissemination  for  high-band 
sensors.  High-band  sensors  are  used  for  Electronic 
Intelligence  (collection  of  pulse  and  constant  wave  radars 
primarily),  while  low-band  refers  to  Communications 


AOC  -  Air  Operations  Center  JIOC  •  Joint  Intelligence  Operations  Center 
AWACS  -  Airborne  Warning  and  JSTARS  -  Joint  Surveillance  Target  Attack 
Control  System  Radar  System 

GH  -  Global  Hawk  JTF  -  Joint  Task  Force 

HQ  -  Headquarters  TARS  -  Tactical/Theater  Airborne 

JFC  -  Joint  Forces  Command  Reconnaissance  System 

Intelligence  (collection  of  radios  of  various  type  and  special 
signals). 

•  AF  DCGS  has  five  core  sites  located  at  Langley  AFB,  Virginia 
(DGS-1);  Beale  AFB,  California  (DGS-2);  Osan  Air  Base, 
Korea  (DGS-3);  Ramstein  Air  Base,  Germany  (DGS-4); 

and  Hickam  AFB,  Hawaii  (DGS-5).  Worldwide,  the 
Air  Force  has  installed  AF  DCGS  at  an  additional  16  sites: 
DGS-Experimental  at  Langley  AFB,  7  Air  National  Guard 
Sites,  and  8  distributed  mission  sites  that  include  deployable 
sites  to  support  special  requirements. 

•  The  DCGS  Integration  Backbone  (DIB)  provides  the 
framework  that  allows  sharing  of  intelligence  services  and  data 
via  web  services.  The  Army,  Navy,  Air  Force,  and  intelligence 
agencies  developed  and  fielded  their  own  versions  of  DCGS, 
which  provide  access  to  the  DIB.  Via  the  DIB,  intelligence 
analysts  can  search  for  and  download  intelligence  information 
and  post  the  intelligence  product  they  produce  for  others  to 
use. 

Mission 

•  The  Air  Force  uses  AF  DCGS  to  connect  to  the  DIB,  manage 
requests  for  sensors,  process  sensor  data,  exploit  sensor  data 
from  multiple  sources,  and  disseminate  intelligence  products. 

•  The  Joint  and  Combined  Force  Air  Component  Commander 
uses  AF  DCGS  to  produce  and  disseminate  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR)  information. 
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•  The  ISR  products  support  intelligence  preparation  of  the  Major  Contractors 

battlespace,  predictive  battlespace  awareness,  indications  •  Raytheon  -  Garland,  Texas 

and  warning,  analysis  of  enemy  courses  of  action,  targeting  •  Lockheed  Martin  -  Denver,  Colorado 

and  weapon  selection,  mission  planning,  and  execution  of  air  •  L-3  Communications  -  Greenville,  Texas 

combat  missions. 


Activity 

•  In  August  and  November  2013,  the  46th  Test  Squadron 
conducted  developmental  and  regression  testing  on  Bulk 
Release  10B.  The  system  demonstrated  major  software 
shortfalls  during  both  of  these  tests  and  the  46th  Test  Squadron 
recommended  against  entering  the  operational  test. 

•  Despite  not  meeting  the  operational  test  entrance  criteria 
(two  CAT  1  and  four  CAT  II  software  deficiencies),  the 
Air  Force  Intelligence,  Surveillance,  and  Reconnaissance 
Agency  (AFISRA)  approved  entrance  into  operational 
testing.  In  January  and  June  2014,  the  605th  TES  conducted 
phases  1  and  2  of  a  two-phase  FDE  to  assess  the  operational 
effectiveness  and  suitability  of  AF  DCGS  Bulk  Release  10B. 

•  In  accordance  with  the  DOT&E  Guidelines  for  Operational 
Test  and  Evaluation  of  Information  and  Business  Systems,  the 
605th  TES  conducted  the  risk  assessment  and  determined  that 
the  FDE  plans  did  not  require  DOT&E  approval  of  the  test 
plan.  The  operational  test  did  not  include  cybersecurity  testing 
by  an  independent  cybersecurity  Red  Team. 

•  On  August  28,  2014,  DOT&E  wrote  a  memorandum  to 
USD(AT&L)  summarizing  the  results  of  the  FDE  and  outlining 
concerns  with  the  progress  of  the  program. 

•  The  AF  DCGS  test  and  evaluation  team  is  working  with 
DOT&E  to  update  the  program’s  Test  and  Evaluation  Master 
Plan. 

•  The  605th  TES  is  working  with  DOT&E  on  the  test  plan  for 
the  next  round  of  GEOINT  upgrades. 

•  In  August  2014,  the  46th  Test  Squadron  conducted 
developmental  and  regression  testing  on  System  Release  3.0. 
The  system  continues  to  exhibit  major  software  shortfalls.  The 
Air  Force  delayed  the  planned  Operational  Utility  Evaluation 
from  June  2014  to  July  2015  to  allow  time  for  the  software 

to  be  fixed  and  demonstrated  in  a  developmental  test  prior  to 
starting  operational  testing. 

Assessment 

•  New  software  delivered  with  Bulk  Release  10B  caused  such 
significant  slowdowns  in  workflow  that  AFISRA  decided  to 
turn  off  the  software  in  order  to  continue  mission  operations. 
Operators  found  the  system  difficult  to  use,  and  the  software 
did  not  meet  reliability  or  availability  requirements. 

•  Although  users  were  able  to  perform  all  necessary  missions 
under  normal  load  conditions,  performance  under  heavy  loads 


could  not  be  determined.  Heavier  loads  are  expected  in  the 
future  when  new  sensors  are  deployed  and  the  number  of 
simultaneous  external  users  are  increased. 

•  The  system  did  not  meet  its  reliability  requirements  because 
of  the  critical  failures  and  downtime.  While  users  can  execute 
their  missions  with  AF  DCGS,  key  software  enhancements  are 
not  maturing.  The  upcoming  FDE  will  test  fixes  to  some  of 
the  software  problems  observed  during  previous  testing. 

•  In  part  because  the  Air  Force  placed  AF  DCGS  in  the 
sustainment  vice  development  phase,  the  program  lacks 
standard  acquisition  processes  and  documentation. 

Specifically,  it  lacks  a  strategy  for  testing  and  evaluation; 
documented  performance  requirements  for  planned 
enhancements;  accurate  software  maturity  trend  information; 
and  an  approved  system  engineering  plan  including  the 
DOD  Architectural  Framework  products. 

•  Survivability  could  not  be  evaluated  due  to  the  lack  of  an 
independent  cybersecurity  test  by  a  certified  Red  Team. 

Recommendations 

•  Status  of  Previous  Recommendations.  DOT&E  previously 
recommended  an  appropriate  level  of  testing  for  the  program; 
the  Air  Force  conducted  adequate  developmental  testing,  but 
did  not  fix  the  problems  discovered  in  developmental  testing 
prior  to  conducting  the  FDE  in  FY14. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1 .  In  the  future,  proceed  to  operational  testing  only  when 
supported  by  successful  development  testing. 

2.  Demonstrate  the  ability  of  AF  DCGS  to  operate  at 
anticipated  workload  levels. 

3.  Complete  a  cybersecurity  assessment  with  a  certified 
Red  Team,  including  operationally  representative  cyber 
attacks. 

4.  Document  the  Air  Force’s  requirements  for  each  delivery 
for  each  of  the  four  programs  and  conduct  adequate  test  and 
evaluation  based  on  a  risk  assessment  in  accordance  with 
DOT&E  guidelines. 

5.  Submit  a  Test  and  Evaluation  Master  Plan  for  DOT&E 
approval,  which  includes  an  accurate  description  of 
AF  DCGS  requirements,  architecture,  and  interfaces 
sufficient  to  justify  the  test  approach. 
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Air  Operations  Center  -  Weapon  System  (AOC-WS) 


Executive  Summary 

•  The  Air  Operations  Center  -  Weapon  System  (AOC-WS)  10.1 
is  a  system-of-systems  that  contains  numerous  third-party 
software  applications,  including  the  Global  Command  and 
Control  System  -  Joint  (GCCS-J),  Theater  Battle  Management 
Core  Systems  -  Force  Level  (TBMCS-FL),  Master  Air 
Attack  Plan  Toolkit  (MAAPTK),  and  Joint  Automated  Deep 
Operations  Coordination  System  (JADOCS). 

•  The  Air  Force  tests  AOC-WS  10. 1  during  a  three-phase 
Recurring  Event  (RE)  test  cycle,  which  includes  event-based 
test  periods  primarily  focused  on  software  upgrades.  The 
software  upgrades  and  associated  test  event  are  designated 
using  similar  terms;  for  example,  AOC-WS  10.1  RE12  is  the 
system  upgrade  tested  during  RE  12. 

-  Phase  1  developmental  testing  is  conducted  at  the 
Combined  Air  Operations  Center  -  Experimental 
(CAOC-X)  at  Langley  AFB,  Virginia. 

-  Phase  2  operational  testing  is  conducted  to  assess 
effectiveness  at  CAOC-X. 

-  Phase  3  operational  testing  is  conducted  at  a  fielded  site  to 
assess  suitability. 

•  In  January  2014,  the  Air  Force  delivered  its  final  report  on 
RE  12  that  inc  luded  the  results  of  Phase  3  operational  testing  at 
612  AOC,  Davis-Monthan  AFB,  Arizona. 

•  AOC-WS  10.1  RE12  has  the  capability  to  produce  the  primary 
products  necessary  to  meet  the  established  AOC  battle  rhythm 
at  threshold  levels.  AOC-WS  10.1  RE  12  demonstrated 
interoperability  with  other  mission-critical  systems.  It 
provides  a  significant  improvement  to  AOCs  in  both  internal 
functionality  and  the  ability  to  interoperate  with  respective 
Combatant  Commands. 

•  AOC-WS  10.1  RE  12  can  be  built,  configured,  and  maintained 
adequately  at  operational  sites  without  the  assistance  of  a 
fielding  team.  Help  desk  support  necessary  to  support  the 
build,  configuration,  and  maintenance  of  AOC-WS  operations 
was  inefficient  and  needs  to  be  improved.  The  duration  and 
nature  of  RE  12  test  events  provided  insufficient  time  to  allow 
DOT&E  to  assess  reliability,  availability,  and  maintainability 
(RAM)  under  operationally  realistic  system  usage. 

•  The  legacy  AOC-WS  10.1  RE  11  has  a  valid  “Authority  to 
Operate”  through  November  2015.  The  AOC-WS  Information 
Assurance  manager  determined  that  the  RE  12  update  has  no 
negative  impact  on  the  AOC-WS  security  posture  and  affirmed 
in  a  memorandum  dated  September  11,  2013,  that  the  existing 
Authority  to  Operate  remains  valid. 

•  The  Air  Force  has  not  yet  used  a  DOD  cyber  Red  Team  to 
fully  assess  cybersecurity  for  AOC-WS  10.1  RE12.  The  Air 
Force  intends  to  accomplish  a  complete  cybersecurity  test  on 
AOC-WS  10.1  during  RE13. 


•  Air  Combat  Command  conducted  a  thorough  analysis  of  the 
three  AOC-WS  10. 1  RE12  outstanding  Category  I  (CAT  I) 
deficiencies  and  accepted  the  risk  of  fielding  AOC-WS  10.1 
RE  12  to  meet  critical  operational  needs;  however,  they  did  so 
while  maintaining  the  expectation  that  the  AOC-WS  Program 
Office  will  fix  those  deficiencies  in  an  expeditious  manner. 
Two  of  the  three  CAT  I  deficiencies  were  re-identified  from 
RE11  and  one  deficiency  discovered  was  new  to  RE12. 

System 

•  The  AOC-WS  is  the  senior  command  and  control  element 
of  the  U.S.  Air  Force’s  Theater  Air  Control  System  and 
provides  operational-level  command  and  control  of  air,  space, 
and  cyberspace  operations,  as  well  as  joint  and  combined 
air,  space,  and  cyberspace  operations.  Capabilities  include 
command  and  control  of  joint  theater  air  and  missile  defense; 
time-sensitive  targeting;  and  Intelligence,  Surveillance,  and 
Reconnaissance  management. 

•  The  AOC-WS  10.1  (AN/USQ-163  Falconer) 
is  a  system-of-systems  that  contains  numerous 
third-party-developed  software  applications  and  commercial 
off-the-shelf  products.  Each  third-party  system  integrated 
into  the  AOC-WS  provides  its  own  programmatic 
documentation. 

•  The  AOC-WS  consists  of: 

-  Commercial  off-the-shelf  hardware 

-  Separate  third-party  software  applications  GCCS-J, 
TBMCS-FL,  MAAPTK,  and  JADOCS,  from  which  the 
AOC-WS  draws  its  capabilities 

-  Additional  third-party  systems  that  accept,  process, 
correlate,  and  fuse  command  and  control  data  from 
multiple  sources  and  share  them  through  multiple 
communications  systems 


AOC-WS 
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•  AOC-WS  10.1  operates  on  several  different  local  area 
networks  (LANs),  including  Secret  Internet  Protocol  Router 
Network,  Joint  Worldwide  Intelligence  Communications 
System,  and  a  coalition  LAN,  when  required.  The  LANs 
connect  the  core  operating  system  and  primary  applications 
to  joint  and  coalition  partners  supporting  the  applicable 
area  of  operation.  Users  can  access  web-based  applications 
through  the  Defense  Information  Systems  Network. 

•  The  Air  Force  tests  AOC-WS  10.1  software  upgrades 
during  REs.  The  Air  Force  refers  to  each  software  upgrade 
by  the  event  during  which  it  was  tested.  For  example, 
AOC-WS  10.1  RE12  is  the  software  upgrade  tested  during 
RE  12. 

•  The  future  AOC-WS  10.2  is  designed  to  deliver  a 
modernized,  integrated,  and  automated  approach  to 
AOC-WS  operations. 


Mission 

The  Commander,  Air  Force  Forces,  or  the  Joint/Combined  Forces 
Air  Component  Commander  use  the  AOC-WS  to  exercise  control 
of  joint  (or  combined)  air  forces  including  planning,  directing, 
and  assessing  air,  space,  and  cyberspace  operations  to  meet 
operational  objectives  and  guidance.  An  operational  AOC  is 
fundamental  in  enabling  centralized  command  and  decentralized 
execution  of  a  theater  air  campaign. 

Major  Contractors 

•  AOC-WS  10.1  Production  Center:  Jacobs  Technology 
Inc.,  Engineering  and  Technology  Acquisition  Support 
Services  -  Hampton,  Virginia 

•  AOC-WS  10.2  Modernization:  Northrop 
Grumman  -  Newport  News,  Virginia 


Activity 

•  The  Air  Force  uses  a  three-phase  RE  test  cycle  for  major 
AOC-WS  10.1  upgrades,  along  with  lower-level  testing  events, 
to  sustain  interoperability  and  cybersecurity  and  provide 
low-risk  upgrades  to  third-party  systems  as  required. 

-  Phase  1  developmental  testing  is  conducted  at  CAOC-X 
Langley  AFB,  Virginia. 

-  Phase  2  operational  testing  is  conducted  at  CAOC-X  to 
assess  effectiveness. 

-  Phase  3  operational  testing  is  conducted  at  a  fielded  site  to 
assess  suitability. 

•  In  August  2013,  the  Air  Force  conducted  operational  testing 
of  AOC-WS  10.1  RE  12.  AOC-WS  10.1  RE12  incorporated 
Defense  Information  Systems  Agency  upgrades  to  GCCS-J, 
updates  to  other  third-party  applications,  and  improvements  to 
the  system’s  cybersecurity  posture. 

•  In  January  2014,  the  Air  Force  completed  its  report  on 
RE  12,  which  included  results  from  Phase  2  operational 
testing  at  CAOC-X,  Langley  AFB,  Virginia,  in  August  2013, 
and  Phase  3  testing  at  612  AOC,  Davis-Monthan  AFB, 
Arizona,  from  November  through  December  2013.  Testing 
at  the  612  AOC  focused  on  the  ability  of  the  install  team  to 
correctly  upgrade  and  configure  the  AOC  from  the  legacy 
AOC-WS  10.1  RE  11  capability  to  the  AOC-WS  10.1  RE12 
configuration,  and  to  perform  backup  and  recovery  actions  on 
AOC-WS  10.1  RE  12. 

•  DOT&E  submitted  an  Interim  Assessment  Memorandum  on 
January  30,  2014,  on  RE12  testing  observed  at  both  CAOC-X 
and  612  AOC.  The  data  from  these  phases  of  testing  formed 
the  basis  for  the  assessment  of  AOC-WS  10.1  RE12’s 
operational  effectiveness  and  suitability. 

•  In  4QFY 14,  the  Air  Force  commenced  the  AOC-WS  10.1 
RE  13  build  at  CAOC-X  in  anticipation  of  beginning  RED 
testing  in  1QFY15. 

•  In  4QFY14,  AOC-WS  10.2  experienced  a  3.5-month  slip 
in  schedule  due  to  the  contractor  not  obtaining  an  Interim 


Authority  to  Test.  Operational  testing  for  AOC-WS  10.2  is 
now  scheduled  to  begin  in  November  2015. 

•  The  Air  Force  conducted  RE  12  testing  in  accordance  with  the 
DOT&E-approved  test  plans. 

Assessment 

•  The  Air  Force  adequately  tested  AOC-WS  10. 1  RED 
through  a  combination  of  developmental  and  operational 
testing;  however,  there  were  significant  known  limitations 
to  cybersecurity  and  RAM  data  collection.  Testing  was 
conducted  in  accordance  with  the  DOT&E-approved  test  plan, 
which  anticipated  the  lack  of  RAM  data.  Therefore,  the  Air 
Force  adopted  a  mitigation  strategy  in  which  they  will  collect 
and  provide  the  required  data  from  fielded  sites,  allowing 
DOT&E  to  refine  the  assessment  results  based  on  the  ongoing 
analysis. 

•  Following  the  completion  of  Phase  3  testing  at  612  AOC,  Air 
Combat  Command  conducted  a  thorough  analysis  of  the  three 
outstanding  CAT  I  Urgent  deficiencies  and  accepted  the  risk 
of  fielding  AOC-WS  10.1  RED  to  meet  critical  operational 
needs,  while  maintaining  the  expectation  that  the  AOC-WS 
Program  Office  will  fix  unresolved  CAT  I  deficiencies  in  an 
expeditious  manner.  This  represents  a  significant  improvement 
to  the  11  open  CAT  I  deficiencies  in  RE11. 

•  RED  successfully  closed  9  of  the  11  RE  11  CAT  I  deficiencies. 
Of  the  two  remaining  CAT  I  deficiencies  from  RE  11,  one 
affected  operational  suitability  and  one  affected  cybersecurity. 
The  third  RED  deficiency  was  a  new  deficiency,  which 
affected  operational  effectiveness. 

•  AOC-WS  10.1  RED  has  the  capability  to  produce  the  primary 
products  necessary  to  meet  the  established  AOC  battle  rhythm 
at  threshold  levels.  AOC-WS  10.1  RE  12  demonstrated 
interoperability  with  other  mission-critical  systems.  It 
provides  a  significant  improvement  to  AOCs  in  both  internal 
functionality  and  the  ability  to  interoperate  with  respective 


264  AOC-WS 


Combatant  Commands.  Four  previously  documented  CAT  I 
deficiencies  in  RE11,  which  were  related  to  GCCS-J  and 
affected  operational  effectiveness,  have  been  closed.  The  new 
RE  12  CAT  1  deficiency  relates  to  audio  deficiencies  in  Defense 
Connect  Online  as  deployed  within  the  AOC-WS.  This 
deficiency  could  negatively  affect  mission-critical  coordination 
activities.  This  deficiency  was  not  previously  discovered 
because  the  operational  site  used  a  thin  client  configuration, 
whereas  the  developmental  testing  used  thick  clients  that  did 
not  exhibit  this  behavior. 

•  AOC-WS  10.1  RE12  can  be  built,  configured,  and  maintained 
adequately  at  operational  sites  without  the  assistance  of  a 
fielding  team.  Help  desk  support  necessary  to  support  the 
build,  configuration,  and  maintenance  of  AOC-WS  operations 
was  inefficient  and  needs  to  be  improved.  This  operational 
help  desk  was  not  utilized  during  developmental  testing,  and 
during  operational  testing,  the  help  desk  had  to  maintain  both 
the  operational  system  and  the  system  under  test,  doubling 
their  workload.  Of  the  five  CAT  I  deficiencies  in  RE1 1 
affecting  operational  suitability,  the  Air  Force  closed  all  but 
one.  The  remaining  deficiency  is  related  to  the  inability  to 
release  Cautions,  Warning,  and  Notes  as  currently  written  to 
coalition  partners;  this  deficiency  does  not  adversely  affect 
U.S.-only  operations. 

•  The  duration  and  nature  of  RE  12  test  events  provided 
insufficient  time  to  allow  DOT&E  to  assess  RAM  under 
operationally  realistic  system  usage.  The  Air  Force  must 
collect  additional  data  at  operational  sites  to  assess  the  effects 
of  RAM  on  AOC  mission  operations.  The  Air  Force  plans 

to  implement  a  technical  RAM  collection  solution  in  the 
modernization  increment,  AOC-WS  10.2. 

•  The  AOC-WS  10.1  RE  12  test  article  and  associated 
documentation  that  entered  OT&E  was  the  direct  output  of  the 
developmental  test-fix-test  cycle.  Time  constraints  precluded 
entering  OT&E  with  a  "clean  rebuild"  of  the  test  article  and  a 
cohesive  consolidation  of  the  documentation  that  incorporated 
all  the  supplements  (software  and  configuration  modifications) 
used  to  "fix"  the  previously  discovered  problems.  Following 
Phase  2  testing,  the  46th  Test  Squadron  conducted  an  RE  12 
regression  build  event  at  Eglin  AFB,  Florida,  that  validated 
that  the  build  process  and  documentation  were  stable  and 
complete  prior  to  proceeding  to  Phase  3  testing  at  612  AOC. 

•  The  legacy  AOC-WS  10.1  RE11  has  a  valid  Authority  to 
Operate  through  November  2015.  The  AOC-WS  Information 
Assurance  manager  determined  that  the  RE  12  update  has  no 
negative  security  impact  on  the  AOC-WS  security  posture  and 
affirmed  in  a  memorandum  dated  September  11,  2013,  that  the 
existing  Authority  to  Operate  remains  valid. 


•  The  Air  Force  has  not  yet  fully  assessed  AOC-WS  10. 1 

RE  12  for  cybersecurity.  AOC-WS  10.1  RE  12  and  recurring 
periodic  software  patches  should  significantly  improve  the 
cybersecurity  posture  of  the  system.  The  Air  Force  intends  to 
accomplish  a  complete  cybersecurity  test  on  AOC-WS  10.1 
during  RE  13. 

•  The  key  to  successful  testing  and  fielding  of  AOC-WS  10.1 
continues  to  be  closer  collaboration  between  the  AOC-WS 
Program  Office  and  Defense  Information  Systems  Agency 
to  ensure  GCCS-J  meets  the  operational  needs  of  the  AOCs. 
Early  AOC-WS  tester  involvement  in  GCCS-J  testing 
continues  to  identify  critical  problems  early  for  corrective 
action. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force  has 
made  progress  in  addressing  the  remaining  two  previous 
recommendations;  however,  cybersecurity  testing  needs 
improvement  and  still  needs  to  be  addressed.  Over  the 
past  two  years,  the  Air  Force  has  increased  its  efforts  with 
two  long-term  FY11  recommendations  (below),  and  this 
engagement  needs  to  continue. 

1 .  Coordinate  with  third-party  programs  to  ensure  that  critical 
AOC-WS  third-party  systems  (such  as  GCCS-J)  have 
testable  requirements  that  meet  AOC-WS  requirements. 

The  requirements  should  be  vetted  within  the  appropriate 
user  and  program  communities  for  schedule  and  funding 
priority. 

2.  Ensure  the  AOC-WS  users  and  test  community  continue 
to  actively  participate  in  GCCS-J  developmental  and 
operational  testing  and  collaborate  to  develop  a  capability  to 
adequately  test  GCCS-J  to  AOC-WS  threshold  stress  levels. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1.  Improve  the  procedures  and  implementation  of  help  desk 
support  to  operational  units  fielding  AOC-WS  10.1  RE  12. 

2.  Conduct  an  assessment  of  operational  risk  to  the  AOC 
warfighting  mission  using  DOD  cyber  Blue  and  Red  Teams 
in  an  operationally  realistic  environment,  consistent  with 
DOT&E  cybersecurity  testing  procedures. 

3.  Require  operational  AOC  sites  to  collect  and  report  all 
significant  RAM  data  to  the  Program  Office,  assess  the 
data  for  needed  system  improvements,  and  report  on 
RAM  improvement  efforts  monthly  to  the  Configuration 
Review  Board.  DOT&E  will  continue  to  review  RAM  data 
periodically  and  adjust  our  findings  in  accordance  with  this 
analysis. 
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Battle  Control  System  -  Fixed  (BCS-F) 


Executive  Summary 

•  The  Air  Force  completed  FOT&E  on  the  Battle  Control 
System  -  Fixed  (BCS-F)  Increment  3,  Release  3.2.2  (R3.2.2) 
at  all  U.S.  air  defense  sites  in  April  2014. 

-  The  BCS-F  R3.2.2  is  operationally  effective  with 
workarounds  and  operationally  suitable,  with  deficiencies 
in  documentation  and  training. 

-  BCS-F  R3.2.2  is  still  not  survivable  against  potential  cyber 
attacks  despite  the  Air  Force’s  efforts  to  improve  R3.2.2’s 
critical  cybersecurity  deficiencies. 

•  All  U.S.  air  defense  sites  were  utilizing  R3.2.2  in 
February  2014.  Upon  completion  of  the  FOT&E,  the 
Air  Force  formally  fielded  R3.2.2. 

System 

•  BCS-F  is  the  tactical  air  battle  management  command  and 
control  system  for  the  two  continental  U.S.  North  American 
Aerospace  Defense  Command  (NORAD)  air  defense  sectors, 
as  well  as  the  Hawaii  and  Alaska  Regional  Air  Operations 
Centers.  The  system  utilizes  commercial  off-the-shelf 
hardware  within  an  open-architecture  software  configuration, 
and  operates  within  the  NORAD  and  U.S.  Pacific  Command 
air  defense  architecture.  The  system  is  employed  by  the  U.S. 
and  Canada. 

•  The  R3.2  upgrade  includes  the  following  system 
enhancements: 

-  Improved  tactical  datalinks  with  additional  Link  16  and 
Link  1 1  message  types  that  enable  the  operators  to  better 
digitally  control  fighters,  send  amplifying  intelligence 
information,  and  create  a  more  comprehensive  air  picture 

-  Air  Tasking  Order  and  Airspace  Control  Order  integration 
with  Theater  Battle  Management  Core  Systems  data 
sources  that  enables  the  operators  to  view  the  most  current 
Air  Tasking  Order/Airspace  Control  Order  and  correlate 
the  information  with  military  aircraft 

-  Data  modification  tools  that  enable  system  administrators 
to  field  changes  to  system  files  and  to  perform  error  checks 
with  greater  fidelity  than  R3. 1 

-  System  control  manager  interface  improvements  that 
enable  the  system  administrator  to  use  improved  system 
performance  monitoring  and  diagnostics 

-  Global  Area  Reference  System  coordinate  conversion  tool 
that  facilitates  a  NORAD  interface  with  global  search  and 
rescue  efforts  by  using  a  common  set  of  coordinates 

-  Remote  Gateway  Manager  control  through  the  virtual 
network  computing  interface  that  provides  the  operators  a 
complete  picture  of  the  available  datalinks  and  flexibility 
to  access  link  information  from  an  operator  workstation 

-  Auxiliary  server  for  offline  training  and  support 
capabilities  at  the  U.S.  air  defense  sectors 


-  Improved  system  capacities  from  10,300  to  15,000  system 
tracks  to  support  single  sector,  continental  U.S.  operations 

•  The  R3.2.2  upgrade  includes  the  following  enhancements: 

-  Ability  to  operate  with  mandatory  International  Civil 
Aviation  Organization  flight  plan  changes 

-  Addition  of  external  firewall/intrusion  detection  system 
sensor 

-  Implementation  of  remote  administrative  management  and 
log  server  capabilities 

-  Audible  and  visual  alert  capabilities  on  the  Computer 
Network  Defense  components 

-  New  network  switch  to  support  the  Information 
Assurance-Demilitarized  Zone  architecture 

-  Newly-designed  protocol  converter  replacing  the  NORAD 
forward  tell  serial  communications  device.  (This  change 
replaces  obsolete  equipment  and  ensures  the  air  picture 
from  the  sector  will  continue  to  be  received  at  NORAD.) 

Mission 

•  NORAD  and  U.S.  Pacific  Command  Commanders  use  BCS-F 
to  execute  command  and  control  and  air  battle  management 
in  support  of  air  sovereignty  and  air  defense  missions  for 
North  American  Homeland  Defense  and  Pacific  Command  air 
defense. 

•  Air  defense  operators  employ  BCS-F  to  conduct  surveillance, 
identification,  and  control  of  U.S.  sovereign  airspace  and 
control  air  defense  assets,  including  fighters,  to  intercept  and 
identify  potential  air  threats  to  U.S.  airspace. 

Major  Contractor 

Thales-Raytheon  Systems  -  Fullerton,  California 
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Activity 

•  The  Air  Force  completed  FOT&E  on  R3.2.2  at  all  U.S. 
air  defense  sites  from  September  2013  to  April  2014.  The 
Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC) 
produced  an  FOT&E  report  on  July  18,  2014. 

•  All  U.S.  air  defense  sites  were  utilizing  R3.2.2  in 
February  2014.  Canadian  air  defense  forces  were  utilizing 
R3.2.2  in  May  2014.  Upon  completion  of  the  FOT&E,  the 
Air  Force  formally  fielded  R3.2.2. 

•  AFOTEC  and  Air  Combat  Command  conducted  operational 
testing  in  accordance  with  a  DOT&E-approved  Test  and 
Evaluation  Master  Plan  and  test  plan. 

•  On  August  13,  2014,  DOT&E  published  a  Major  Automated 
Information  System  FOT&E  report  on  BCS-F  R3.2.2 

Assessment 

•  R3.2.2  resolved  two  of  the  five  major  operational  effectiveness 
deficiencies  associated  with  battle  management  discovered 
during  R3.2  and  R3.2.0.1  operational  testing.  Additionally, 
FOT&E  of  R3.2.2  revealed  four  new  deficiencies  associated 
with  battle  management  operations.  Operator  workarounds 
mitigated  these  deficiencies  to  an  acceptable  level. 

•  R3.2.2  is  operationally  suitable,  although  the  Air  Force  did 
not  collect  sufficient  operational  test  data  to  demonstrate 
the  availability  and  reliability  requirements  with  statistical 
confidence. 

-  During  952.35  hours  of  testing,  R3.2.2  did  not  experience 
any  critical  failures  or  downtime.  One  critical  failure 
occurred  after  system  fielding  resulting  in  two  hours 

of  system  downtime.  Including  this  failure  in  total 
system  operating  time  during  the  FOT&E  resulted  in 
an  operational  availability  of  99.99  percent  (80  percent 
confidence  intervals  are  99.78  and  99.99  percent). 

-  Additionally,  as  of  September  30,  2014,  the  system  has 
operated  at  all  four  U.S.  air  defense  sector  sites  without 
any  additional  critical  failures  since  February  2014. 

This  equates  to  over  23,000  hours  (5,800  hours  at  each 
of  four  sites)  with  only  one  critical  failure.  The  system 
requirement  for  Mean  Time  Between  Critical  Failure 
(MTBCF)  is  greater  than  or  equal  to  10,000  hours. 

Current  data  indicate  with  a  66.9  percent  confidence  that 
the  MTBCF  requirement  has  been  met. 

•  R3.2.2  was  maintainable  for  routine  maintenance  actions, 
but  the  observed  8.6  hour  Mean  Time  Between  Corrective 
Maintenance  Action  (MTBCMA)  did  not  meet  the  100-hour 
requirement.  This  was  not  a  critical  shortfall  since  the 
maintenance  actions  had  no  negative  effect  on  operations  or 
operator  workload. 


•  While  R3.2.2  is  operationally  suitable,  technical 
documentation  and  training  for  the  system  administrators  was 
deficient. 

•  R3.2.2  remains  deficient  in  all  cybersecurity  assessment  areas. 
The  system  is  poorly  equipped  to  detect,  protect,  react,  and 
restore/recover  from  attacks  by  current  cyber  threats  despite 
the  fact  that  R3.2.2  was  designed  to  resolve  many  critical 
cybersecurity  deficiencies.  The  Air  Force  plans  to  address 
some  of  the  outstanding  cybersecurity  deficiencies  through 
implementation  of  the  Computer  Network  Defense  Service 
Provider  agreement  in  1QFY15. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 
satisfactorily  addressed  all  but  three  of  the  previous 
recommendations.  The  Air  Force  still  needs  to: 

1.  Correct  and  formalize  all  BCS-F  Increment  3  system 
documentation  and  training  deficiencies. 

2.  Develop  a  plan  for  remote  workstation  management 
to  include  sustainment,  training,  documentation,  and 
Information  Assurance  compliance. 

3.  Upgrade  the  System  Support  Facility  to  support  a  more 
robust  BCS-F  developmental  and  operational  testing 
capability  in  order  to  minimize  the  impact  of  overall  testing 
at  the  operational  sites. 

•  FY14  Recommendations.  The  Air  Force  should: 

1 .  Correct  the  three  remaining  operational  effectiveness 
deficiencies  discovered  during  R3.2  and  R3.2.0.1  testing, 
as  well  as  the  four  new  deficiencies  associated  with  battle 
management  discovered  during  R3.2.2. 

2.  Correct  and  formalize  all  BCS-F  R3.2.2  documentation  and 
training  deficiencies. 

3.  Improve  reliability  to  meet  the  threshold  requirement  for 
MTBCMA. 

4.  Fully  assess  system  vulnerabilities  and  correct  identified 
cyber  deficiencies. 

5.  Re-evaluate  BCS-F  survivability  against  cyber  attacks  after 
the  Computer  Network  Defense  Service  Provider  has  been 
implemented.  This  evaluation  is  scheduled  to  occur  in 
1QFY15. 

6.  Implement  appropriate  policies,  procedures,  and  tools 
for  system  administrators  to  effectively  respond  to 
unauthorized  intrusions. 

7.  Correct  network  configuration  deficiencies  to  more 
effectively  detect  unauthorized  intrusions. 
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Defense  Enterprise  Accounting  and  Management 

System  (DEAMS) 


Executive  Summary 

•  The  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  conducted  an  operational  assessment  (OA)  of 
the  Defense  Enterprise  Accounting  and  Management  System 
(DEAMS)  from  August  2013  through  February  2014.  The 
test  locations  included  Fleadquarters  U.S.  Transportation 
Command  (F1Q  USTRANSCOM)  and  Fleadquarters  Air 
Mobility  Command  (F1Q  AMC),  both  located  at  Scott  AFB, 
Illinois;  the  Defense  Finance  and  Accounting  Service  (DFAS) 
in  Limestone,  Maine;  and  base  operations  at  five  AMC  bases. 

•  In  conjunction  with  the  OA,  the  177th  Information  Aggressor 
Squadron  conducted  a  Cyber  Economic  Vulnerability 
Assessment  (CEVA)  under  the  oversight  of  DOT&E.  The 
assessment  determined  it  would  be  possible  for  knowledgeable 
adversaries  to  penetrate  DEAMS  and  conduct  fraudulent 
activities,  including  exploiting  DEAMS  for  economic  gain. 

•  DEAMS  demonstrated  progress  toward  resolving  previous 
effectiveness  shortfalls  and  the  OA  found  few  new  systemic 
defects,  but  some  balances  are  still  inaccurate.  The  quality  of 
reports,  responses  to  ad  hoc  queries,  and  usefulness  of  error 
messages  remained  poor. 

•  DEAMS  demonstrated  progress  toward  resolving  previous 
suitability  problems.  The  system  met  its  reliability  and 
availability  requirements,  but  it  experienced  problems  with 
cybersecurity,  training,  timely  resolution  of  software  defects, 
and  interoperability  with  critical  systems. 

•  The  DEAMS  IOT&E  has  been  delayed  due  to  performance 
issues. 

System 

•  DEAMS  is  a  Major  Automated  Information  System  that 
uses  commercial  off-the-shelf  Enterprise  Resource  Planning 
software  to  provide  accounting  and  management  services 
for  the  Air  Force  and  U.S.  Transportation  Command 
(USTRANSCOM). 


DEAMS 


Manage  DoD 
Appropriated  & 
Working  Capital 
Funds 


Process  Budgetary, 
Accounting,  & 
Vendor  Pay 
Transactions 


General  Ledger 

Funds  Control 

Accounts  payable/receivable 

Cost  Accounting/Management 

Billings/Collections 

Commitments/Obligations 

Purchasing  &  Receipts 

Revenue 

Analysis/Decision  Support 

Budget  Formulation 

Budget  Execution 

Cost  Modeling 

Provide  Financial 
Data  to  Decision 
Makers 


Provide  Budget 
Formulation,  Funds 
Distribution,  and 
Cost  Modeling 


•  DEAMS  operates  on  the  Global  Combat  Support  System  -  Air 
Force  Integration  Framework.  It  interfaces  with  approximately 
40  other  systems  that  provide  travel,  payroll,  disbursing, 
transportation,  logistics,  acquisition,  and  accounting  support. 
DEAMS  will  deploy  to  USTRANSCOM,  the  Air  National 
Guard,  and  all  Air  Force  major  commands,  including  those 
overseas.  There  are  expected  to  be  nearly  30,000  users 
worldwide  by  2017. 

Mission 

•  USTRANSCOM  and  Air  Force  financial  managers  use 
DEAMS  to  compile  and  share  accurate,  up-to-the-minute 
financial  management  data  and  information  across 
USTRANSCOM  and  the  Air  Force. 

•  USTRANSCOM,  Air  Force,  and  DOD  leadership  use  DEAMS 
to  access  vital,  standardized,  real-time  financial  data  and 
information  to  make  strategic  business  decisions  and  to  satisfy 
congressional  and  DOD  requirements  for  auditing  funds, 
standardizing  financial  ledgers,  timely  reporting,  and  reduction 
of  costly  rework. 

Major  Contractor 

Accenture  Federal  Services  -  Dayton,  Ohio 


Activity 

•  In  August  20 1 4,  AFOTEC  observed  developmental  testing  of 
DEAMS  Release  3. 

•  AFOTEC  conducted  a  DEAMS  OA  from  August  2013 
through  February  2014.  The  test  locations  included 
HQ  USTRANSCOM  and  HQ  AMC,  both  located  at 

Scott  AFB,  Illinois;  the  DFAS  facility  in  Limestone,  Maine; 
and  base  operations  at  five  AMC  bases. 

•  In  conjunction  with  the  OA,  the  177th  Information  Aggressor 
Squadron  conducted  a  CEVA  under  the  oversight  of  DOT&E. 


•  DOT&E  issued  a  report  on  the  DEAMS  OA  on  June  10, 
2014. 

•  AFOTEC  began  the  DEAMS  IOT&E  in  October  2014 
in  accordance  with  a  DOT&E-approved  test  plan. 

Only  the  end-of-year  activities  were  observed.  The 
Program  Executive  Office  delayed  certification  of  the 
program  for  the  bulk  of  IOT&E  due  to  performance  issues. 
IOT&E  is  scheduled  to  resume  in  January  2015. 
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Assessment 

•  During  the  OA,  DEAMS  demonstrated  progress  toward 
resolving  previous  effectiveness  shortfalls  and  the  OA  found 
few  new  systemic  defects.  Some  balances  are  still  inaccurate 
and  the  quality  of  reports,  responses  to  ad  hoc  queries,  and 
usefulness  of  error  messages  remain  poor. 

-  DEAMS  now  prevents  the  posting  of  transactions  that 
exceed  control  targets.  While  maintenance  of  fiscal 
accountability  in  balancing  funds  for  the  General  Ledger 
and  subsidiary  accounts  (a  Key  Performance  Parameter)  is 
still  unsatisfactory,  it  has  improved. 

-  Nine  of  11  tested  measures  that  quantify  the  quality  of 
DEAMS  financial  information  now  meet  thresholds,  and 
the  other  two  (standards  compliance  and  General  Ledger 
operations)  are  improving.  All  three  timeliness  thresholds 
(recording  transactions,  vendor  payments,  and  period-end 
closings)  were  met  and  user  access  roles  are  generally 
being  assigned  properly. 

-  DEAMS  Release  3  includes  a  new  software  application  for 
producing  reports;  it  will  be  evaluated  during  the  IOT&E. 

•  The  DEAMS  OA  also  demonstrated  progress  toward  resolving 
previous  suitability  issues.  The  system  met  its  reliability 

and  availability  requirements,  but  continues  to  suffer  from 
problems  with  training,  timely  resolution  of  software  defects, 
interoperability  with  critical  systems,  and  cybersecurity  issues. 

-  Software  reliability  growth,  as  measured  by  reduction  of 
defect  reports,  has  been  static.  Although  configuration 
management  has  improved,  there  are  still  a  large  number 
of  unresolved  defects.  Several  required  capabilities  and 
enhancements  are  scheduled  for  near-term  release  and  are 
being  tested  during  IOT&E. 

-  One  critical  interface  with  the  Departmental  Cash 
Management  System  (which  manages  and  reconciles 
cash  disbursements,  reimbursement,  collections,  and 
receipts)  did  not  meet  interoperability  requirements.  Not 
all  interfaces  were  tested  during  the  OA.  All  deployed 
critical  interfaces  must  be  operationally  demonstrated  in 
conjunction  with  IOT&E. 


-  New  user  training  was  inadequate  because  it  focused  on 
how  to  navigate  within  DEAMS  and  did  not  provide  users 
with  a  real  understanding  of  the  system  and  its  application 
to  their  day-to-day  work  processes. 

•  The  CEVA  demonstrated  that  a  cyber  adversary  with 
knowledge  of  DEAMS  and  DOD  business  practices  could 
gain  access  to  DEAMS  and  conduct  denial-of-service  and 
fraudulent  activities,  including  exploiting  DEAMS  for 
economic  gain.  Further  information  can  be  found  in  the 
classified  appendix  to  DOT&E’s  June  2014  OAreport. 

•  DOT&E  will  prepare  a  report  on  the  DEAMS  IOT&E  upon 
completion  of  testing. 

Recommendations 

•  Status  of  Previous  Recommendations.  TheFY12 
recommendation  for  the  Program  Office  and  Functional 
Management  Office  to  document  workarounds  at  the  base 
level  remains  unresolved.  The  Air  Force  has  made  progress  on 
all  FY13  recommendations.  The  recommendation  to  correct 
Information  Assurance  deficiencies  and  perform  Information 
Assurance  and  financial  fraud  penetration  testing  was  partially 
satisfied  by  conducting  the  CEVA;  both  cooperative  and 
adversarial  penetration  testing  will  be  conducted  during 
IOT&E  to  further  assess  cybersecurity  vulnerabilities.  The 
Air  Force  also  made  progress  toward  updating  training 
materials  during  the  OA;  training  will  be  assessed  again 
during  IOT&E.  The  recommendation  to  provide  more  on-site 
technical  support  to  new  users  at  the  base  level  has  been 
addressed. 

•  FY 14  Recommendations.  The  Program  Office  should: 

1.  Accelerate  the  correction  of  balance  accuracy  defects 
in  order  to  meet  the  Accurate  Balance  of  Funds  Key 
Performance  Parameter  and  achieve  full  auditability. 

2.  Address  the  cybersecurity  recommendations  provided  in 
the  classified  CEVA  appendix  to  DOT&E’s  June  2014  OA 
report. 
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F-15E  Radar  Modernization  Program  (RMP) 


Executive  Summary 

•  The  Air  Force  Operational  Test  and  Evaluation  Center 
conducted  IOT&E  from  March  through  September  2013  to 
assess  the  system’s  operational  effectiveness,  suitability,  and 
mission  capability. 

•  DOT&E  published  a  classified  report  on  the  operational 
effectiveness,  suitability,  and  mission  capability  of  the 
F-15E  Radar  Modernization  Program  (RMP)  system  upon 
completion  of  IOT&E  data  analysis.  The  system  entered 
full-rate  production  in  March  2014. 

•  IOT&E  results  demonstrated: 

-  The  F-15E  RMP  is  operationally  effective  with  significant 
improvements  in  air-to-air  capabilities. 

-  RMP  operating  modes  and  pilot-vehicle  interfaces  are 
functionally  equivalent  to  those  of  the  legacy  APG-70 
radar. 

-  APG-82(V)1  hardware  operational  reliability  and 
maintainability  support  F-15E  operational  availability 
requirements. 

-  Additional  monitoring  of  system  suitability  will  be 
required  in  order  to  fully  assess  system  reliability  and 
maintainability  metrics  due  to  the  limited  flight  hours  and 
low  failure  rate  observed  during  IOT&E. 

•  At  the  conclusion  of  the  IOT&E,  there  were  unresolved 
RMP  system  supportability  and  deployability  shortfalls.  The 
Air  Force  has  yet  to  resolve  long-term  support  equipment 
provisioning,  functionality,  and  overall  APG-82(V)1  spares 
posture  necessary  to  support  the  deployment  of  RMP-equipped 
F-15E  operational  squadrons. 

System 

•  The  F-15E  is  a  twin-engine,  tandem-seat,  fixed-wing, 
all-weather,  multi-role  fighter  aircraft.  The  F-15E  has  a 
fully-missionized  cockpit  and  a  multimode  air  intercept 
and  air-to-ground  radar,  giving  the  aircrew  the  capability 
to  employ  air-to-air  and  air-to-ground  munitions,  a  20  mm 
cannon,  and  countermeasures  for  evading  enemy  fire. 

•  The  RMP  replaces  the  F-15E  legacy  APG-70 
mechanically-scanned  radar  with  an  active 
electronically-scanned  array  (AESA)  system  designated  the 
APG-82(V)1.  The  RMP  is  designed  to  retain  functionality 
of  the  legacy  radar  system  while  providing  expanded  mission 
employment  capabilities  to  include: 

-  Near-simultaneous  interleaving  of  selected  air-to-air  and 
air-to-ground  functions 

-  Enhanced  air-to-air  and  air-to-ground  combat  identification 
capabilities 

-  Longer  range  air-to-air  target  detection  and  enhanced  track 
capabilities 


-  Longer  range  and  higher  resolution  air-to-ground  radar 
mapping 

-  Improved  ground  moving  target  track  capability 

•  The  RMP  upgrade  is  also  intended  to  address  legacy  F-15E 
radar  system  suitability  shortfalls  including:  poor  reliability, 
parts  obsolescence,  and  high  sustainment  costs.  The  Air  Force 
intends  to  retrofit  the  RMP  across  the  existing  F-15E  fleet. 

•  The  RMP  APG-82(V)1  design  leverages  capabilities  from 
currently  fielded  AESA  radar  systems.  The  APG-82(V)1 
antenna  and  power  supply  are  currently  in  use  on  the  F-15C 
APG-63(V)3  program,  and  the  radar  receiver/exciter  and 
Common  Integrated  Sensor  Processor  are  based  on  the 
F/A-18E/F  APG-79  AESA  system. 

•  Other  hardware  and  software  modifications  comprising  the 
RMP  effort  include  a  more  powerful  Environmental  Control 
System,  updates  to  the  aircraft  Operational  Flight  Program  and 
Electronic  Warfare  software,  a  new  radio  frequency  tunable 
filter,  and  aircraft  modifications  to  include  a  new  wideband 
radome  and  wiring  changes. 

Mission 

A  unit  equipped  with  the  F-15E  conducts  all  weather,  day  and 

night  missions  to  include: 

•  Offensive  and  Defensive  Counterair 

•  Conventional  Air  Interdiction  and  Nuclear  Strike 

•  Close  Air  Support  and  Strike  Coordination  and 
Reconnaissance 

•  Suppression  of  Enemy  Air  Defenses 

•  Combat  Search  and  Rescue 

Major  Contractors 

•  The  Boeing  Company  -  St.  Louis,  Missouri 

•  Raytheon  -  El  Segundo,  California 


F-15E  RMP 
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Activity 

•  The  Air  Force  Operational  Test  and  Evaluation  Center 
conducted  the  F-15E  RMP  IOT&E,  which  completed  in 
September  2013,  in  accordance  with  the  DOT&E-approved 
Test  and  Evaluation  Master  Plan  and  IOT&E  plan. 

•  In  March  2014,  DOT&E  published  a  classified  report  on  the 
operational  effectiveness,  suitability,  and  mission  capability  of 
the  F-15E  RMP  system  upon  completion  of  the  IOT&E  data 
analysis. 

•  The  F-15E  RMP  entered  full-rate  production  in  March  2014. 

Assessment 

•  DOT&E  assesses  that  the  F-15E  RMP: 

-  Is  operationally  effective  and  operating  modes  and 
pilot-vehicle  interfaces  are  functionally  equivalent  with 
those  of  the  legacy  APG-70  radar  system. 

-  Provides  significantly  improved  capability  in  the  air-to-air 
operational  environment  compared  to  that  of  the  legacy 
APG-70  radar  system. 

-  Demonstrated  comparable  air-to-ground  radar  performance 
compared  with  that  of  the  legacy  system  and  improvements 
in  target  location  accuracy. 

-  Software  stability  did  not  meet  the  Air  Force  Mean 
Time  Between  Software  Anomaly  criteria  of  30  hours 
during  IOT&E.  However,  post-IOT&E  flight  testing  of 
a  subsequent  radar  software  version  corrected  the  single 
anomaly  that  resulted  in  6  of  12  observed  software  stability 
events  encountered  in  IOT&E. 

-  Hardware  operational  reliability  and  maintainability 
support  F-15E  operational  availability  requirements. 
However,  limited  flight  hours  and  the  low  failure  rate 
observed  throughout  the  evaluation  period  precluded 
DOT&E’s  ability  to  confirm,  with  confidence,  that  the 
APG-82(V)1  hardware  reliability,  maintenance  man-hours 
per  flight  hour,  mean  repair  time,  and  built-in  test  fault 
diagnostics  requirements  were  met.  Therefore,  additional 
monitoring  of  system  suitability  will  be  required  in  order 
to  fully  assess  system  performance  in  these  areas. 


•  At  the  conclusion  of  the  IOT&E,  there  were  unresolved  RMP 
system  supportability  and  deployability  shortfalls  to  include: 

-  The  Air  Force  currently  lacks  a  long-term  programmatic 
solution  for  providing  ground-cooling  carts  to  service  the 
APG-82(V)1  at  operational  unit  locations. 

-  The  Gore®  communications  cables  that  connect  the 
radar  Common  Integrated  Signal  Processor  to  the 
receiver/ exciter  cannot  be  functionally  checked  with  the 
Joint  Services  Electronic  Combat  Systems  Tester. 

-  The  Air  Force  has  yet  to  define  the  Readiness  Spares 
Package  provisioning  necessary  to  determine  the  number 
of  436L  pallets  and  Gore®  communications  cables  needed 
to  support  the  deployment  of  RMP-equipped  operational 
F-15E  squadrons. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force  is 
addressing  RMP  software  stability  issues  previously  identified 
in  the  FY12  and  FY13  Annual  Report. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1.  Correct  the  anomalies  identified  in  IOT&E  that  resulted  in 
software  reliability  events  to  ensure  F-15E  RMP  software 
stability  meets  Air  Force  requirements. 

2.  Provide  a  long-term  solution  for  APG-82(V)1  ground 
cooling  carts,  Gore®  communications  cable  spares  posture 
and  ground  test  set  compatibility,  and  Readiness  Spares 
Package  provisioning  and  deployment  pallet  posture,  in 
order  for  the  F-15E  RMP  system  to  be  fully  supportable  and 
deployable. 

3.  Continue  to  monitor  installed  system  reliability,  availability, 
and  maintainability  metrics  to  confirm,  with  confidence, 
that  APG-82(V)1  hardware  reliability,  maintenance 
man-hours  per  flight  hour,  mean  repair  time,  and  built-in 
test  fault  diagnostics  performance  meet  the  Air  Force 
requirements. 


272 


F-15E  RMP 


F-22A  Advanced  Tactical  Fighter 


Executive  Summary 

•  F-22A  Increment  3.2A  is  a  software-only  modernization 
effort  integrating  Link  16  Receive,  enhanced  Combat 
Identification,  and  enhanced  Electronic  Protection  capabilities. 
Increment  3. 2 A  developmental  testing  proceeded  throughout 
FY14.  Software  stability  and  radar  performance  shortfalls 
discovered  late  in  developmental  testing  precluded  the  start 
of  FOT&E  planned  by  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  for  FY14.  This  FOT&E  is 
currently  projected  to  begin  in  FY 15. 

•  F-22A  Modernization  Increment  3.2B,  a  separate  Major 
Defense  Acquisition  Program,  achieved  Milestone  B  in 
June  2013.  Laboratory  and  flying  test  bed  developmental 
testing  continued  throughout  FY14.  IOT&E  is  planned  for 
FY17. 

•  AIM-120D  weapons  models  required  for  Increment  3.2B 
IOT&E  were  not  on  contract  to  support  planned  FY17 
IOT&E.  Should  these  models  not  be  available  to  meet 
planned  IOT&E,  operational  testing  will  be  delayed,  or 
additional  live-fire  missile  events  beyond  those  already 
projected  may  be  required  during  IOT&E. 

System 

•  The  F-22A  is  an  air-superiority  fighter  that  combines  low 
observability  to  threat  radars,  sustained  high  speed,  and 
integrated  avionics  sensors. 

•  Low  observability  reduces  threat  capability  to  engage  F-22As 
with  current  adversary  weapons. 

•  The  aircraft  maintains  supersonic  speeds  without  the  use  of  an 
afterburner. 

•  Avionics  that  fuse  information  from  the  Active  Electronically 
Scanned  Array  radar,  other  sensors,  and  datalinked  information 
for  the  pilot  enable  employment  of  medium-  and  short-range 
air-to-air  missiles,  guns,  and  air-to-ground  munitions. 

•  The  Air  Force  designed  the  F-22A  to  be  more  reliable  and 
easier  to  maintain  than  legacy  fighter  aircraft. 

•  F-22A  air-to-air  weapons  are  the  AIM-120C  radar-guided 
missile,  the  AIM-9M  infrared-guided  missile,  and  the  M61A1 
20  mm  gun. 

•  F-22A  air-to-ground  precision  strike  capability  consists  of  the 
1,000-pound  Joint  Direct  Attack  Munition  and  the  250-pound 
Small  Diameter  Bomb  (SDB)  Increment  One. 

•  The  F-22A  program  delivers  capability  in  increments. 
Incremental  Enhanced  Global  Strike  modernization  efforts 
include  the  following  current  and  projected  increments: 

-  Increment  3.1  provides  enhanced  air-to-ground  mission 
capability,  to  include  geolocation  of  selected  emitters, 
electronic  attack,  air-to-ground  synthetic  aperture 


radar  mapping  and  designation  of  surface  targets,  and 
SDB  integration.  Increment  3.1  is  currently  fielding  in 
operational  F-22A  units. 

-  Increment  3.2A  is  a  software-only  upgrade  intended 
to  provide  improved  electronic  protection.  Link  16 
Receive,  and  Combat  Identification  capabilities  in  FY15. 
Increment  3.2A  is  a  modernization  effort  within  the 
scope  of  the  F-22  A  Advanced  Tactical  Fighter  baseline 
acquisition  program  of  record. 

-  Increment  3.2B  is  a  separate  Major  Defense  Acquisition 
Program  modernization  effort  intended  to  integrate 
AIM-120D  and  AIM-9X  missile  systems  and  provide 
additional  electronic  protection  enhancements 

and  improved  emitter  geolocation  capability.  The 
Increment  3.2B  IOT&E  is  currently  planned  for  FY17. 

Mission 

A  unit  equipped  with  the  F-22  A: 

•  Provides  air  superiority  over  friendly  and  non-permissive, 
contested  enemy  territory 

•  Defends  friendly  forces  against  fighter,  bomber,  or  cruise 
missile  attack 

•  Escorts  friendly  air  forces  into  enemy  territory 

•  Provides  air-to-ground  capability  for  counter-air, 
strategic-attack,  counter-land,  and  enemy-air  defense 
suppression  missions 

Major  Contractor 

Lockheed  Martin  Aeronautics  Company  -  Fort  Worth,  Texas 
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Activity 

•  The  Air  Force  conducted  F-22A  testing  in  accordance  with 
the  DOT&E-approved  Test  and  Evaluation  Master  Plan. 

•  F-22A  Increment  3.2A  developmental  testing  proceeded 
throughout  FY14.  Software  stability  and  radar  performance 
shortfalls  discovered  late  in  developmental  testing 
precluded  the  start  of  AFOTEC’s  planned  FY14  FOT&E. 
Additional  unanticipated  software  releases  required  further 
developmental  testing,  and  FOT&E  is  scheduled  to  begin  in 
1QFY15. 

•  F-22A  Modernization  Increment  3.2B  achieved  Milestone  B 
in  June  2013.  Post  Milestone  B,  F-22  Increment  3.2B 
developmental  testing  continued  throughout  FY14.  IOT&E  is 
planned  for  FY17. 

•  At  the  conclusion  of  the  FY12  F-22A  FOT&E,  the  Air  Force 
reduced  the  level  of  support  needed  to  sustain  the  Air-to-Air 
Range  Infrastructure  (AAR1)  capability  and  ensure  system 
readiness  for  subsequent  F-22A  OT&E.  In  FY14,  the 

Air  Force  undertook  efforts  to  restore  the  system  to  support 
the  planned  FY14  Increment  3.2A  FOT&E. 

Assessment 

•  F-22  Increment  3.2A  realized  software  stability  and  radar 
performance  shortfalls  late  in  the  developmental  flight  test 
schedule.  These  shortfalls  necessitated  additional  unplanned 
software  releases  in  order  to  demonstrate  readiness  for 
FOT&E.  Accordingly,  the  FOT&E  planned  for  3QFY14  was 
postponed  pending  resolution  of  the  shortfalls  and  completion 
of  developmental  test  and  evaluation.  FOT&E  is  now 
projected  to  begin  in  1QFY15. 

•  F-22  Increment  3.2B  requires  upgraded  threat  and  weapons 
models  for  IOT&E  effectiveness  evaluation  trials  that  will  be 
performed  in  both  open-air  flight  test  at  the  Nevada  Test  and 
Training  Range  and  in  the  Lockheed  Martin  F-22  Advanced 
Combat  Simulator  (ACS)  in  Marietta,  Georgia.  At  the  end  of 
FY14,  AIM-120D  modeling  was  not  yet  on  contract  to  support 
FY17  IOT&E.  Should  the  models  not  be  available  for  FY17 
ACS  trials,  IOT&E  will  be  delayed. 

•  F-22A  OT&E  requires  the  use  of  the  AARI  instrumentation 
system  for  flight  test  missions  at  the  Nevada  Test  and  Training 
Range. 


-  AARI  is  designed  to  enable  testers  to  credibly  shape  air 
battles  and  resolve  complex  operational  mission  outcomes 
through  real-time  instrumented  air  and  surface  threat 
engagements. 

-  The  system  aids  real-time,  open-air  threat  and  friendly 
force  removal  assessments,  and  is  required  for  F-22A 
OT&E  flight  test  adequacy.  AARI  mission  outcomes 
further  serve  as  a  foundation  for  ACS  accreditation  for 
F-22A  OT&E  effectiveness  evaluations  performed  at  the 
ACS. 

-  The  Air  Force  struggled  to  ensure  AARI  readiness  to 
support  planned  FY14  F-22A  Increment  3.2A  testing. 

At  the  conclusion  of  FY12  F-22A  testing,  the  Air  Force 
began  an  extensive  AARI  network  upgrade  and  the 
implementation  of  new  weapons  models  to  support  future 
F-22A  and  F-35  operational  testing.  However,  the  level 
of  effort  the  Air  Force  placed  on  maintaining  AARI 
functionality  was  insufficient  to  ensure  readiness  for 
Increment  3.2A  FOT&E,  and  AARI  system  test  readiness 
experienced  unplanned  delays.  To  ensure  readiness  for 
FY15  and  beyond  F-22A  operational  testing,  the  Air  Force 
will  need  to  fully  support  development,  modernization,  and 
sustainment  of  the  AARI  system. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 

continues  to  address  all  previous  recommendations. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1.  Continue  to  resolve  F-22  Increment  3.2A  software 
anomalies  and  radar  performance  shortfalls  in 
developmental  testing  before  proceeding  to  formal 
AFOTEC  FOT&E  in  FY15. 

2.  Resolve  AARI  sustainment,  test  readiness,  and 
modernization  shortfalls  in  order  to  support  both  near-term 
F-22  Increment  3.2A  and  future  Increment  3.2B  IOT&E 
test  adequacy. 

3.  Commit  sufficient  resources  necessary  to  ensure  that 
AIM-120D  models  are  available  for  F-22  Increment  3.2B 
FY17  IOT&E. 
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Family  of  Advanced  Beyond  Line-of-Sight  Terminals 

(FAB-T) 


Executive  Summary 

•  In  2012,  the  USD(AT&L)  directed  the  Air  Force  to  award 
a  contract  to  a  second  source  to  compete  with  the  Boeing 
Company  for  development  of  the  Family  of  Advanced  Beyond 
Line-of-Sight  Terminals  (FAB-T)  Command  Post  Terminal 
(CPT).  The  Air  Force  awarded  a  development  contract  to  the 
Raytheon  Corporation. 

•  On  June  2,  2014,  the  Air  Force  down-selected  from  the 
two  development  contractors  and  awarded  the  Raytheon 
Corporation  the  production  contract  for  the  CPT.  The 
Air  Force  plans  to  exercise  production  contract  options  to 
produce  CPT  terminals  after  a  successful  Milestone  C  decision. 

•  The  program’s  CPT  development  is  approximately  four  months 
behind  Raytheon’s  originally  proposed  schedule.  Continued 
delays  may  be  experienced  due  to  a  highly-concurrent 
developmental  test  schedule. 

•  The  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  conducted  an  operational  assessment  from 
July  through  December  2014,  based  on  contractor  and 
government  developmental  flight  testing  to  inform  the 
Milestone  C  decision  planned  for  March  2015. 


System 

•  FAB-T  consists  of  ground  and  aircraft  communication 
terminals  with  two  terminal  types  -  CPTs  and  Force  Element 
Terminals  (FETs).  FAB-T  is  part  of  the  terminal  and  control 
segments  of  the  Advanced  Extremely  High  Frequency 
(AEHF)  satellite  system  and  is  designed  to  operate  with 
AEHF  Low  Data  Rate  (75  -  2,400  bits  per  second  (bps))  and 
Extended  Data  Rate  (up  to  8.192  Megabits  per  second  (Mbps)) 
waveforms. 

•  The  CPT  is  intended  to  replace  existing  airborne  (E-4B 
and  E-6B),  ground-fixed,  and  ground-transportable  Milstar 
command  post  terminals.  The  CPT  will  include  satellite  and 
network  control  functions,  end-user  telecommunication  device 
interfaces,  and  the  ability  to  operate  the  terminal  from  a  distant 
location  using  a  remote  node. 

•  The  FET  is  intended  to  be  installed  in  airborne  force  elements 
(B-2,  B-52,  and  RC-135).  The  FET  is  a  requirement  but  is 


currently  neither  funded  nor  on  contract  for  development  and 
production. 

Mission 

•  The  President,  the  Secretary  of  Defense,  Combatant 
Commanders,  and  support  Air  Force  component  forces  will 
use  FAB-T  to  provide  strategic  nuclear  and  non-nuclear 
command  and  control  with  EHF,  wideband,  protected,  and 
survivable  communications  terminals  for  beyond  line-of-sight 
communications. 

•  U.S.  Strategic  Command  will  use  the  FAB-T  to  perform  the 
satellite  Telemetry,  Tracking,  and  Commanding  functions  of 
the  AEHF/Milstar  constellation,  including  management  of  the 
satellites,  communication  networks  and  cryptologic  keys. 

Major  Contractor 

Raytheon  Net-Centric  Systems  -  Marlborough,  Massachusetts 


Activity 

•  In  2012,  the  USD(AT&L)  directed  the  Air  Force  to  award 
a  contract  to  a  second  source  to  compete  with  the  Boeing 
Company  for  development  and  production  of  the  CPT  with 
Presidential  and  National  Voice  Conferencing  (PNVC). 
Additionally,  USD(AT&L)  directed  the  Air  Force  to  defer 


development  of  the  FET.  The  Air  Force  awarded  the  second 
sourcing  contract  for  the  CPT  development  to  the  Raytheon 
Corporation. 

•  The  Defense  Information  Systems  Agency  is  developing  the 
PNVC  equipment  that  FAB-T  will  use  to  provide  strategic 
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services  to  Senior  Leadership,  which  is  a  Key  Performance 
Parameter. 

•  The  FAB-T  program  manager  planned  to  conduct  contractor 
and  government  developmental  flight  testing  for  the  Boeing 
variant  from  February  through  June  2013.  Schedule  delays 
and  developmental  regression  testing  extended  this  time 
through  April  2014. 

•  The  Raytheon  Corporation  conducted  two  system 
demonstrations  in  2013  to  provide  the  government  confidence 
they  had  a  viable  solution  that  could  interoperate  with  the  EFLF 
satellites  and  terminals. 

•  On  June  2,  2014,  the  Air  Force  down-selected  from  the 
two  development  contractors  and  awarded  the  Raytheon 
Corporation  the  production  contract  for  the  CPT.  The 
Air  Force  plans  to  exercise  production  contract  options 
to  produce  CPT  terminals  after  a  successful  Milestone  C 
decision. 

•  AFOTEC  compressed  their  normal  planning  timeline  and 
rapidly  developed  an  operational  assessment  test  plan 

to  support  the  FAB-T  program  manager’s  new  schedule 
based  on  the  Raytheon  selection.  DOT&E  approved  the 
operational  assessment  test  plan  on  July  17,  2014.  AFOTEC 
conducted  an  operational  assessment  in  accordance  with  the 
DOT&E-approved  test  plan  from  4QFY14  -  2QFY15  based 
on  contractor  and  government  developmental  flight  testing  to 
inform  the  Milestone  C  decision  projected  for  March  2015. 


•  With  the  selection  of  Raytheon  as  the  producer  of  the  FAB-T 
CPT,  the  schedule  and  developmental  test  strategy  have 
significantly  changed.  Consequently,  the  previous  Test  and 
Evaluation  Master  Plan  (TEMP)  based  on  the  Boeing  variant 
is  no  longer  valid.  The  Air  Force  is  working  to  develop  a  new 
CPT  Milestone  C  TEMP  that  supports  the  current  test  program. 

Assessment 

•  The  Defense  Information  Systems  Agency’s  PNVC 
development  requires  an  Engineering  Change  Proposal 
because  current  variants  do  not  meet  environmental  conditions 
sufficient  for  aircraft  operations.  This  increases  schedule  risk 
in  providing  aircraft-variant  production  units  for  FOT&E 
projected  for  FY17. 

•  The  program’s  CPT  development  is  approximately  four 
months  behind  Raytheon’s  original  proposed  schedule. 
Continued  delays  may  be  experienced  due  to  a  highly 
concurrent  developmental  test  schedule. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force  has 
addressed  the  four  previous  recommendations. 

•  FY 14  Recommendation. 

1 .  The  Air  Force  should  expedite  staffing  of  the  FAB-T  TEMP 
in  order  to  meet  the  proposed  Milestone  C  date. 
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Global  Positioning  System  (GPS)  Enterprise 


Executive  Summary 

•  The  Air  Force  conducted  significant  development  for  all  three 
enterprise  segments  in  2014,  including  component  testing  for 
GPS  III  and  prototype  risk-reduction  testing  for  Military  GPS 
User  Equipment  (MGUE)  Increment  1,  but  did  not  conduct 
any  operational  testing  for  the  GPS  enterprise  in  2014. 

•  Expected  operational  testing  dates  for  all  three  segments  have 
changed  from  those  listed  in  the  current  Enterprise  Test  and 
Evaluation  Master  Plan  (ETEMP)  approved  in  March  2012. 

•  Significant  delays  to  the  Next  Generation  Operational  Control 
System  (OCX)  pose  risks  to  the  Air  Force’s  ability  to  sustain 
the  operational  GPS  constellation,  as  the  Air  Force  may 
require  operational  use  of  GPS  III  satellites  before  OCX 
Block  1  is  available  to  control  those  satellites.  The  Air  Force 
is  developing  plans  for  a  contingency  operations  capability. 

•  Other  concerns  include: 

-  OCX  delays  add  risk  that  OT&E  will  not  discover  GPS  III 
satellite  deficiencies  until  it  is  too  late  to  correct  them. 

-  The  Air  Force’s  current  plan  for  operational  assessment 
of  the  MGUE  Increment  1  will  not  provide  sufficient 
data  to  support  an  informed  Milestone  C  decision  by 
USD(AT&L),  adding  risk  to  that  decision. 

-  Air  Force  and  Army-requested  MGUE  Increment  1  Lead 
Platform  changes  will  reduce  the  pathfinding  value  of  Lead 
Platform  testing,  and  the  Air  Force’s  proposed  MGUE 
Increment  1  source  selection  approach  may  reduce  post 
IOT&E  competition  and  delay  fielding  of  MGUE  to 
non-Lead  Platforms. 

-  The  Air  Force  is  not  mitigating  several  significant  risks  to 
the  GPS  enterprise,  including  the  potential  loss  of  a  critical 
industrial  production  capability,  the  unfunded  need  for 
overseas  monitoring  stations,  indications  of  receiver-host 
platform  integration  and  compatibility  problems,  and 
sustainment  of  the  de  facto  27-satellite  operational 
constellation. 

-  Air  Force  GPS  enterprise  schedules  provided  to  DOT&E 
and  other  OSD  components  are  not  accurate,  current,  or 
consistent  with  GPS  segment  schedules. 

-  The  Air  Force  has  requested  that  USD(AT&L)  waive 
the  requirement  for  the  MGUE  Increment  1  operational 
assessment  (OA).  Failure  to  conduct  an  OA  prior  to  the 
Milestone  C  decision  for  MGUE  Increment  1  would  add 
significant  risk  to  the  program. 

System 

•  The  GPS  enterprise  is  an  Air  Force-managed,  satellite-based 
radio  navigation  system-of-systems  that  provides  worldwide 
military  and  civil  users  accurate  position,  velocity,  and  time. 


•  The  current  GPS  enterprise  consists  of  three  operational 
segments: 

-  Space  Segment  -  The  GPS  spacecraft  constellation 
consists  of  a  minimum  of  24  operational  satellites  in 
semi-synchronous  orbit.  The  Air  Force  has  successfully 
launched  over  65  GPS  satellites  and  currently  operates 
over  30  healthy  GPS  satellites,  comprised  of  Block  IIA 
(launched  1990-1996),  Block  HR  (1997-2004), 

Block  IIR-M  (2005-2009),  and  Block  IIF  (2010-present). 

-  Control  Segment  -  The  GPS  control  segment  consists  of 
primary  and  backup  GPS  master  control  stations,  satellite 
control  antennas,  a  pre-launch  satellite  compatibility 
station,  and  geographically-distributed  operational 
monitoring  stations.  The  current  GPS  control  segment,  the 
Operational  Control  System  (OCS)  supports  (1)  operation 
of  GPS  Block  IIF  and  legacy  satellites,  (2)  Selective 
Availability/ Anti-Spoof  Module  capabilities  in  GPS 

User  Equipment,  and  (3)  Launch/Early  Orbit,  Anomaly 
Resolution,  and  Disposal  Operations. 

-  User  Segment  -  There  are  many  versions  of  military  GPS 
mission  receivers  fielded  on  a  multitude  of  operational 
systems  and  combat  platforms,  including  the  most 
common  Defense  Advanced  GPS  Receivers  and  embedded 
Ground-Based  GPS  Receiver  Application  Modules 
(GB-GRAM),  numbering  in  the  hundreds  of  thousands. 

•  In  2000,  the  DOD  approved  initiation  of  a  GPS  enterprise 
modernization  effort  to  include  upgrades  to  all  three 
segments,  along  with  new  civil  and  military  signals 
(M-code).  In  addition  to  replenishment  of  the  constellation, 
this  modernization  is  intended  to  improve  both  military 
and  civil  signal  integrity  and  service  quality  in  terrain-  and 
geography-impeded  environments,  as  well  as  in  the  presence 
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of  unintentional  and  deliberate  interference.  Modernized  GPS 
enterprise  improvements  include: 

-  Space  Segment  -  GPS  III  satellites,  an  Acquisition 
Category  (ACAT)  ID  program,  have  a  design  life  that  far 
exceeds  that  of  all  earlier  blocks.  GPS  III  will  be  capable 
of  transmitting  a  fourth  civil  signal  and  higher-powered 
M-code,  as  well  as  all  legacy  military  and  civil  navigation 
signals  of  previous  satellite  blocks. 

-  Control  Segment  -  OCX,  an  ACAT  ID  program,  replaces 
the  current  OCS/ Architecture  Evolution  Plan  control 
segment  and  is  backward  compatible  with  Block  HR  and 
later  satellites.  OCX  will  provide  (1)  control  of  GPS  III 
satellites  and  legacy  signals,  (2)  full  control  of  modernized 
civil  and  M-code  signals,  and  (3)  significant  cybersecurity 
improvements  over  OCS. 

-  User  Segment  -  MGUE  Increments  1  and  2  are  pre-Major 
Defense  Acquisition  Programs  expected  to  be  ACAT  ID. 
MGUE  Increment  1  includes  the  Ground-Based  GPS 
Receiver  Application  Module-Modernized  (GB-GRAM-M) 
form  factor  for  ground  and  low-dynamic  platforms 

such  as  small  unmanned  aircraft  systems,  and  the  GPS 
Receiver  Application  Module-Standard  Electronic 
Module-E/Modernized  (GRAM-S/M)  for  maritime  and 
aviation  applications.  MGUE  Increment  2  requirements 
are  in  development  and  presumed  to  address  requirements 
and  applications  not  addressed  by  MGUE  Increment  1, 
including  handheld,  precision  guided  munition,  and 
standard  space  receiver  applications. 

Mission 

•  GPS  is  a  global  utility,  integral  to  U.S.  national  security, 
economic  growth,  transportation  safety,  homeland  security, 


and  the  world’s  economic  infrastructure.  It  is  U.S.  national 
policy  to  provide  continuous  worldwide  access  to  GPS,  for 
peaceful  civil  uses,  and  to  employ  GPS  to  satisfy  U.S.  civil  and 
national  security  needs. 

•  Combatant  Commanders,  U.S.  military  forces,  allied 
nations,  and  various  civilian  agencies  rely  on  GPS  to  provide 
highly-accurate,  real-time,  all-weather,  and  time  information  to 
operational  users  worldwide.  GPS  provides  force  enhancement 
for  combat  operations  and  military  forces  in  the  field  on  a  daily 
basis  throughout  a  wide  variety  of  global  strategic,  operational, 
and  tactical  missions. 

•  Properly  equipped  military  forces  will  employ  modernized  GPS 
capabilities  to  (1)  determine  or  contribute  to  their  determination 
of  their  location  and  velocity,  (2)  support  precision  munitions 
targeting  and  employment,  and  (3)  synchronize  operations  and 
secure  communications  in  all  environments. 

Major  Contractors 

•  Space  Segment 

-  Block  IIR/IIR-M/III  satellites:  Lockheed  Martin  Space 
Systems  -  Denver,  Colorado 

-  Block  IIF  satellites:  Boeing  Space  and  Intelligence 
Systems  -  Seal  Beach,  California 

•  Control  Segment 

-  OCS:  Lockheed  Martin  -  Colorado  Springs,  Colorado 

-  OCX:  Raytheon  Company,  Intelligence,  Information  and 
Services  -  Aurora,  Colorado 

•  User  Segment  (MGUE  Inc  1) 

-  L-3  Communications/Interstate  Electronics 
Corporation- Anaheim,  California 

-  Rockwell  Collins  -  Cedar  Rapids,  Iowa 

-  Raytheon  Company  -  El  Segundo,  California 


Activity 

•  The  Air  Force  conducted  significant  development  for  all  three 
enterprise  segments  in  2014,  including  component  testing 
for  GPS  III  and  prototype  risk-reduction  testing  for  MGUE 
Increment  1,  but  did  not  conduct  any  operational  testing  for 
the  GPS  enterprise  in  2014. 

•  Expected  operational  testing  dates  for  all  three  segments  have 
changed  from  those  listed  in  the  current  ETEMP  approved 

in  March  2012,  as  indicated  below.  Those  schedule  changes 
result  from  development  and  delivery  delays  to  GPS  III  and 
OCX,  and  from  Air  Force-proposed  changes  to  the  MGUE 
Increment  1  Acquisition  Strategy,  as  well  as  from  Air  Force 
and  Army -proposed  changes  to  their  Service-nominated  Lead 
Platforms  for  MGUE  Increment  1. 

•  The  Air  Force  currently  expects  to  conduct  operational  tests 
for  each  GPS  segment  as  follows: 

-  OA  of  MGUE  Increment  1  accelerated  from  late  2016  to 
late  2015,  to  support  USD(AT&L)’s  combined  Milestone 
B/C  decision  for  MGUE  Increment  1,  under  an  accelerated 
Air  Force-proposed  schedule. 


-  IOT&E  of  MGUE  Increment  1  accelerated  from  2021  to 
2017,  to  support  USD(AT&L)’s  Beyond  Low-Rate  Initial 
Production  decision  for  MGUE  Increment  1,  leading 

to  procurement  and  fielding  of  MGUE  Increment  1 
components  by  the  Services,  for  their  respective  platforms. 

-  The  Operational  Utility  Evaluation  of  OCX  Block  I  and 
GPS  III  satellite  vehicle  (SV)  01  slipped  from  early  2016 
to  early  2019,  supporting  an  Air  Force  fielding  decision  for 
OCX  Block  1  and  operational  acceptance  of  GPS  III  SV01. 

-  Multi-service  OT&E  of  the  modernized  GPS  enterprise 
in  2020,  including  OCX  Block  II  and  all  associated 
navigation  warfare  and  modernized  signal  and  messaging 
functions  testing,  supporting  an  Air  Force  fielding  decision 
for  OCX  Block  II. 

•  The  next  revision  of  the  GPS  ETEMP  is  in  coordination  within 
the  Air  Force  and  with  Service  Operational  Test  Agencies; 
DOT&E  expects  to  receive  the  revised  ETEMP  in  early  2015 
for  OSD  approval,  and  it  should  describe  developmental  T&E 
(DT&E)  and  OT&E  to  take  place  in  2015  and  beyond.  The 


278 


GPS  Enterprise 


ETEMP  will  require  further  revision  to  incorporate  updated 
development  and  test  planning  assumptions  entailed  by 
changes  to  all  three  GPS  enterprise  segment  schedules,  and 
to  describe  OT&E  for  any  contingency  capability  developed 
to  operate  GPS  III  satellites  prior  to  the  availability  of  OCX 
Block  I. 

Assessment 

•  No  OT&E  test  data  are  available  at  this  point. 

•  DOT&E’s  November  2014  memorandum  to  USD(AT&L) 
identified  several  significant  concerns  regarding  sustainment 
of  the  GPS  enterprise  and  execution  of  GPS  enterprise 
modernization. 

-  Impact  of  OCX  delays  on  GPS  III  OT&E.  OCX  delays 
add  risk  that  OT&E  will  not  discover  GPS  III  deficiencies 
until  it  is  too  late  to  correct  them.  OCX  Block  1  is  needed 
to  operate  and  support  OT&E  of  GPS  III  satellites.  The 
schedule  delays  for  OCX  Block  I  delivery  will  delay 
OT&E  of  GPS  III  until  after  at  least  six,  and  as  many  as 
eight,  GPS  III  satellites  have  been  built  and  launched.  This 
introduces  significant  risk  that  effectiveness  and  suitability 
deficiencies  in  GPS  III  satellites  will  not  be  discovered 
until  it  is  too  late  to  prevent  their  introduction  to  the 
operational  constellation. 

-  Risk  to  the  MGUE  Increment  1  OA.  The  Air  Force’s 
current  plan  for  the  MGUE  Increment  1  OA  will  not 
provide  sufficient  data  to  support  an  informed  Milestone  C 
decision  by  USD(AT&L),  adding  risk  to  that  decision. 

■  The  Air  Force  has  proposed  an  accelerated  acquisition 
strategy  for  MGUE  Increment  1 ,  which  combines 
Milestones  B  and  C,  and  eliminates  the  engineering, 
manufacturing,  and  development  phase,  requiring 
earlier  execution  of  the  OA  for  MGUE  Increment  1 . 

The  Air  Force  does  not  currently  intend  to  integrate 
MGUE  Increment  1  onto  any  Lead  Platforms  prior 

to  the  scheduled  OA  in  2015.  DOT&E  informed  the 
Air  Force  in  December  2013  that  an  adequate  OA  for 
MGUE  Increment  1  should  encompass  integration  and 
developmental  test  of  at  least  one  Lead  Platform  per 
MGUE  form  factor  being  evaluated.  This  shortfall  will 
significantly  reduce  the  OA’s  utility  in  informing  the 
Milestone  C  decision. 

■  The  Air  Force  MGUE  risk-reduction  activities  to-date 
with  the  RQ-11B  Raven  small  unmanned  aircraft  system 
have  had  mixed  results,  which  underscore  substantial 
remaining  challenges  for  MGUE  integration  on  other 
platforms,  and  reinforce  the  need  for  an  adequate  OA. 

-  Lead  Platform  Changes  and  Source  Selection.  Air  Force 
and  Army-requested  Lead  Platform  changes  will  reduce 
the  pathfinding  value  of  Lead  Platform  testing,  and  the 
Air  Force’s  proposed  MGUE  Increment  1  source  selection 
approach  may  reduce  post  IOT&E  competition  and  delay 
fielding  of  MGUE  to  non-Lead  Platforms. 

■  MGUE  Increment  1  Lead  Platforms  are  pathfinders, 
intended  to  represent  the  operational  environment  and 
integration  challenges  for  all  DOD  platforms  using  those 


respective  form  factors.  The  four  Service-nominated 
Lead  Platforms  identified  in  the  MGUE  Increment  1 
Capabilities  Development  Document  are: 

-  The  RQ-11B  Raven  (Army)  and  Joint  Light  Tactical 
Vehicle  (Marine  Corps)  for  the  GB-GRAM-M  form 
factor 

-  The  F-15E  (Air  Force)  and  DDG-51,  Arleigh  Burke 
class  destroyer  (Navy)  for  the  GRAM-S/M  form 
factor. 

■  The  U.S.  Army  and  Air  Force  recently  requested  to 
replace  their  designated  Lead  Platforms  with  the  Defense 
Advanced  GPS  Receiver  Distributed  Device-equipped 
Stryker  family  of  vehicles  and  the  B-2  Spirit, 
respectively.  This  changes  the  range  of  integration 
challenges  and  operational  factors  that  can  be  evaluated 
prior  to  a  Beyond  Low-Rate  Initial  Production  decision, 
and  may  reduce  the  degree  to  which  the  Lead  Platforms 
are  able  to  “pathfind”  for  non-Lead  Platforms.  In  turn, 
this  may  increase  the  integration  challenges  and  OT&E 
requirements  for  non-Lead  Platforms  and  ultimately 
delay  fielding  of  MGUE  Increment  1  receivers  for  the 
rest  of  the  DOD. 

■  The  Air  Force-proposed  MGUE  Increment  1  Acquisition 
Strategy  source  selection  approach  entails  each  Service 
selecting  only  a  single  vendor  solution,  from  among 

the  multiple  vendor  solutions  that  may  be  certified  and 
available,  to  integrate  with  their  Lead  Platforms  and 
undergo  OT&E.  There  are  two  possible,  undesirable 
outcomes  to  this  strategy. 

-  The  two  lead  platforms  for  each  MGUE  form  factor 
choose  the  same  vendor.  This  leads  to  adequate 
operational  testing  of  the  single  vendor  chosen  but 
inadequate  operational  testing  of  any  other  MGUE 
vendor  solutions,  reducing  post-IOT&E  competition. 

-  The  two  lead  platforms  for  each  MGUE  form  factor 
choose  different  vendors.  This  leads  to  inadequate 
operational  testing  for  each  MGUE  vendor  solution 
because  the  IOT&E  will  not  have  tested  any  solution 
across  the  widest  possible  range  of  operational 
environments  and  integration  challenges  for  that 
form  factor.  This  could  result  in  higher  integration 
costs  and  will  drive  expanded  OT&E  for  non-Lead 
Platforms  that  subsequently  integrate  MGUE 
Increment  1. 

-  Inadequate  Articulation  of  Risks.  The  Air  Force  is  not 
mitigating  several  significant  problems  as  risks  to  the  GPS 
enterprise.  Examples  include: 

■  The  potential  sale  to  foreign  ownership  of  the  only 
U.S.  trusted  foundry  producing  the  application-specific 
integrated  circuits  for  MGUE  Increment  1,  which 
would  impede  development  and  production  of  MGUE 
Increment  1. 

■  The  descoped  and  unprogrammed  provision  of  M-code 
monitoring  stations  outside  the  continental  U.S.,  which 
would  impede  M-code  anomaly  detection  and  resolution. 
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■  Observed  MGUE  Increment  1  thermal,  power,  and 
interface  implementation,  which  may  adversely  affect 
integration  with  host  platforms. 

■  Risk  to  the  probability  of  sustaining  availability  of  a 
27-satellite  operational  constellation,  which  has  become 
the  de  facto  expected  standard  of  service  for  military 
GPS  users  due  to  its  benefit  for  terrain-impeded  users. 

-  Inaccurate,  Implausible,  and  Incoherent  Schedules. 

Air  Force  schedules  for  the  GPS  enterprise,  provided  in 
support  of  OSD  acquisition  decisions,  reflect  timelines  for 
integration  and  test  of  MGUE  Increment  1  that  have  not 
been  endorsed  by  the  responsible  Lead  Platform  program 
offices  or  operational  test  agencies.  Independent  and  OSD 
reviews  have  identified  a  lack  of  accuracy,  currency,  and 
coherence  in  GPS  enterprise  schedules,  which  negatively 
affects  effective  program  management  and  oversight. 

-  Overstatement  of  MGUE  Development  Maturity.  The 
Air  Force  requested  in  November  2014  that  USD(AT&L) 
waive  the  requirements  for  Critical  Design  Review  and 
O  A  of  MGUE  Increment  1 ,  on  the  premise  of  technical 
maturity  in  the  MGUE  Increment  1  program.  Based  on  the 
results  of  risk-reduction  activities  to  date,  and  historical 
experience  with  complex  and  challenging  host  platform 
integration,  MGUE  Increment  1  has  not  demonstrated 
technical  maturity  that  would  warrant  waiver  of  these 
critical  assessment  functions. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force  has 
partially  addressed  the  five  previous  recommendations  listed  in 
the  2011  Annual  Report: 

1.  There  has  been  no  opportunity  thus  far  for  end-to-end 
testing  of  OCX  with  MGUE  receivers,  but  the  ETEMP 
incorporates  planning  for  the  Multi-service  OT&E  of  the 
modernized  GPS  enterprise,  which  will  address  end-to-end 
testing.  The  Air  Force  is  not  yet  planning  for  adequate 
integration  on  representative  platforms  to  enable  timely 
OT&E  in  representative  environments  in  support  of 
well-informed  acquisition  and  fielding  decisions.  The 

Air  Force  should  continue  to  plan  for  end-to-end  testing 
of  the  GPS  enterprise,  including  integration  on  Lead 
Platforms,  and  DT&E  and  OT&E  in  realistic  operational 
environments,  in  time  to  support  acquisition  decisions. 

2.  Synchronization  of  the  development  of  the  Space,  Control, 
and  User  segments  has  only  marginally  improved. 
Descriptions  of  the  impact  of  delays  in  each  segment  upon 
the  GPS  enterprise  and  other  segment  schedules  are  often 
not  clearly  articulated.  The  Air  Force  should  ensure  that 
status  and  critical  interdependencies  of  each  enterprise 
segment  are  well  understood,  and  shoidd  promptly  assess 


and  disseminate  to  all  stakeholders  those  predicted 
enterprise  impacts  resulting  from  forecast  changes  in 
segment  schedules. 

3.  The  revised  ETEMP  now  in  Service  coordination  reflects 
clear  improvements  in  planning  for  comprehensive  and 
realistic  cybersecurity  testing  of  the  GPS  enterprise, 
although  additional  revisions  will  be  necessary  to  reflect 
GPS  segment  changes  and  DOT&E’s  August  2013 
guidance,  Procedures  for  Operational  Test  and  Evaluation 
of  Cybersecurity  in  Acquisition  Programs.  The  Air  Force 
should  continue  to  refine  its  cybersecurity  testing  approach 
to  GPS. 

4.  The  Services  have  made  progress  in  emphasizing/enforcing 
the  use  of  crypto-keyed  GPS  receivers,  but  should  redouble 
their  efforts. 

5.  The  Services  have  made  progress  in  developing  concepts 
of  operations  and  tactics,  techniques,  and  procedures  for 
keying  GPS  receivers,  but  that  has  not  translated  into  use  of 
encrypted  receivers  for  all  military  operations. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1 .  Mitigate  the  risks  to  the  GPS  enterprise  associated  with 
delays  to  OCX  delivery  and  the  inability  to  conduct  OT&E 
of  GPS  III  SV01  prior  to  the  launch  of  as  many  as  eight 
GPS  III  satellites. 

2.  Integrate  and  conduct  DT&E  of  MGUE  Increment  1  on  at 
least  one  Lead  Platform  per  form  factor  in  time  to  support 
an  OA  informing  MGUE  Increment  1  Milestone  C. 

3.  Continue  the  engineering,  manufacturing,  and  development 
of  MGUE  Increment  1  until  it  has  demonstrated  maturity 
in  both  functional  performance  and  integration  with  host 
platforms. 

4.  Assess  the  degree  to  which  designated  Lead  Platforms  for 
MGUE  Increment  1  cover  the  range  of  operational  factors 
and  integration  challenges  for  the  complete  portfolio  of 
DOD  platforms  each  MGUE  form  factor  is  intended  to 
support. 

5.  Ensure  each  available  MGUE  Increment  1  vendor  solution 
for  a  given  form  factor  is  integrated  with  all  Lead  Platforms 
for  that  respective  form  factor  to  support  adequate  MGUE 
IOT&E. 

6.  Identify  and  articulate  a  mitigation  plan  of  action  and 
milestones  for  all  significant  risks  to  the  GPS  enterprise. 

7.  Maintain  and  disseminate  coherent,  accurate,  and  timely 
schedule  information  for  all  segments  of  the  GPS 
enterprise,  ensuring  that  each  segment  schedule  and  the 
enterprise  master  schedule  reflect  interdependencies 
between  segments.  Ensure  these  segment  and  enterprise 
schedules  reflect  the  most  current  government  estimates  and 
are  caveated  to  reflect  any  un-validated  assumptions. 
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KC-46A 


Executive  Summary 

•  DOT&E  approved  the  post-Milestone  B  Test  and  Evaluation 
Master  Plan  (TEMP)  in  January  2013.  The  TEMP  approval 
memorandum  identified  planned  test  program  shortfalls  that 
require  resolution  prior  to  the  Milestone  C  TEMP  approval. 

•  Readiness  for  the  scheduled  start  of  IOT&E  continues  to  be 
high-risk  with  a  12-month  delay  expected.  It  is  unlikely  that 
Boeing  and  the  Air  Force  will  develop  a  schedule  that  delivers 
18  different  certified  receiving  aircraft,  as  described  in  the 
Milestone  B  TEMP,  prior  to  the  start  of  familiarization  training 
for  the  IOT&E. 

•  The  program  has  made  advances  in  collecting  and  analyzing 
live  fire  test  data  needed  to  address  the  KC-46A  vulnerability 
to  threat-induced  dry  bay  fires,  threat-induced  wing  structural 
failures,  and  flight  deck  armor  effectiveness. 

System 

•  The  KC-46A  aerial  refueling  aircraft  is  the  first  increment  of 
replacement  tankers  (179)  for  the  Air  Force’s  fleet  of  KC-135 
tankers  (more  than  400). 

•  The  KC-46A  design  uses  a  modified  Boeing  767-200ER 
commercial  airframe  with  numerous  military  and 
technological  upgrades,  such  as  the  fly-by-wire  refueling 
boom,  the  remote  air  refueling  operator’s  station,  additional 
fuel  tanks  in  the  body,  and  defensive  systems. 

•  The  KC-46A  is  intended  to  provide  boom  (pictured)  and 
probe-drogue  refueling  capabilities.  The  Air  Force  intends 
to  equip  the  KC-46A  with  an  air-refueling  receptacle  so  that 
it  can  also  receive  fuel  from  other  tankers,  including  legacy 
aircraft. 

•  The  KC-46A  is  designed  to  have  significant  palletized  cargo 
and  aeromedical  capacities;  chemical,  biological,  radiological, 
nuclear  survivability;  and  the  ability  to  host  communications 
gateway  payloads. 

•  Survivability  enhancement  features  are  incorporated  into  the 
KC-46A  design. 

-  Susceptibility  is  reduced  with  an  Aircraft  Survivability 
Equipment  suite  consisting  of  Large  Aircraft  Infrared 


Countermeasures,  the  ALR-69A  Radar  Warning  Receiver 
(RWR),  and  a  Tactical  Situational  Awareness  System.  The 
suite  is  intended  to  compile  threat  information  from  the 
RWR  and  other  on-  and  off-board  sources  and  prompt  the 
crew  with  an  automatic  re-routing  suggestion  in  the  event 
of  a  threat. 

-  Vulnerability  is  reduced  through  the  addition  of  fuel  tank 
inerting  and  integral  armor  to  provide  some  protection  to 
the  crew  and  critical  systems. 

Mission 

Commanders  will  use  units  equipped  with  the  KC-46A  to: 

•  Perfonn  air  refueling  to  accomplish  six  primary  missions 
to  include  nuclear  operations  support,  global  strike  support, 
air  bridge  support,  aircraft  deployment,  theater  support,  and 
special  operations  support.  Secondary  missions  will  include 
airlift,  aeromedical  evacuation,  emergency  aerial  refueling,  air 
sampling,  and  support  of  combat  search  and  rescue. 

•  Operate  in  day/night  and  adverse  weather  conditions  over  vast 
distances  to  support  U.S.,  joint,  allied,  and  coalition  forces. 

•  Operate  in  a  non-permissive  environment. 

Major  Contractor 

The  Boeing  Company,  Commercial  Aircraft  in  conjunction  with 

Defense,  Space  &  Security  -  Seattle,  Washington 


Activity 

•  The  KC-46A  Integrated  Test  Team  met  quarterly  from 
April  2011  through  June  2014.  The  September  2014  meeting 
was  postponed  pending  resolution  of  design  and  contractual 
issues. 

•  DOT&E  approved  the  post-Milestone  B  TEMP  in 
January  2013,  with  caveats. 

•  DOT&E  approved  the  Air  Force  Operational  Test  and 
Evaluation  Center’s  plan  for  a  KC-46A  operational  assessment 
to  support  the  Milestone  C  decision. 


•  Developmental,  operational,  and  Federal  Aviation 
Administration  test  planning  is  in  progress.  The  contractor’s 
preliminary  Stage  4  (final  build)  test  plans  were  submitted 
to  the  Air  Force.  More  than  90  percent  of  the  plans  were 
accepted. 

•  Initial  flight  of  the  first  Engineering  and  Manufacturing 
Development  (EMD)  aircraft  is  more  than  five  months  late, 
primarily  due  to  electrical  wiring  design  complexity.  The 
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KC-46Ahas  approximately  60  percent  more  wiring  than 
Boeing’s  commercial  767-200  aircraft. 

•  Test  venues  for  the  ALR-69A  RWR  and  the 

AAQ-24  Large  Aircraft  Infrared  Countermeasures  system  have 
been  agreed  to  by  all  parties.  Detailed  test  planning  for  each 
venue  remains  to  be  completed. 

•  The  Air  Force  planned  and  executed  three  major  live  fire  test 
series:  (1)  Center  Wing  Dry  Bay  Fire  Vulnerability,  (2)  Wing 
Flydrodynamic  Ram  Evaluation,  and  (3)  Armor  Effectiveness. 

•  Boeing  is  preparing  draft  detailed  test  plans  for 
Electromagnetic  Pulse  (EMP)  testing  based  on  the 
contract-specified  design  margin  of  6  decibels  (dB),  which 
provides  a  higher-risk  hardening  approach. 

Assessment 

•  DOT&E  identified  shortfalls  in  the  planned  test  program  that 
require  resolution  prior  to  Milestone  C  TEMP  approval.  The 
TEMP  requires  increased  detail  in  a  number  of  areas.  The 
Air  Force  is  addressing  shortfalls  in  the  planned  test  program. 

-  Mitigation  is  needed  for  the  impact  of  concurrent  activities 
and  planned  flying  hours  for  the  EMD  program  that  place  a 
high  demand  on  limited  aircraft  and  simulator  resources. 

-  The  operational  test  aircrew  and  maintenance  personnel 
must  have  time  to  attain  their  training  requirements  and 
establish  proficiency  in  operationally  representative 
conditions  before  the  start  of  IOT&E. 

-  The  technical  order  verification  process  must  be  completed 
before  the  start  of  IOT&E. 

-  Sufficient  calendar  time  must  be  allotted  for  correction 
of  discrepancies  and/or  deficiencies  discovered  during 
developmental  testing  prior  to  the  planned  start  of 
operational  testing. 

•  The  Air  Force  is  continuing  to  analyze  existing  schedule 
risks  and  potential  mitigations.  DOT&E  analysis  of  Boeing 
schedules  with  regard  to  aerial-refueling  certifications, 
aircraft  and  support  equipment  technical  orders,  and 
operator/ maintainer  training  indicates  that  operational  testing 
will  likely  slip  at  least  12  months.  It  is  unlikely  that  Boeing 
and  the  Air  Force  will  develop  a  schedule  that  delivers 

18  different  certified  receiving  aircraft,  as  described  in  the 
Milestone  B  TEMP,  prior  to  the  start  of  training  for  the 
IOT&E.  Also,  until  all  detailed  test  plans  (known  as  Stage  4 
test  plans)  are  approved,  DOT&E  will  not  have  sufficient 
insight  to  determine  if  there  are  adequate  mitigations  to  reduce 
the  schedule  risks  prior  to  the  start  of  IOT&E. 

•  The  test  team  is  working  a  cybersecurity  strategy  to  be 
consistent  with  DOT&E  guidance;  however;  specific  details  to 
execute  an  adequate  test  are  not  yet  defined. 

•  The  first  flight  of  the  initial  KC-46A  aircraft  scheduled  for 
January  2015  is  now  planned  for  April  2015.  This  delay  will 
alter  the  planned  certification  schedule  of  Phase  I  receiving 
aircraft  and  may  delay  the  August  2015  Milestone  C  decision. 

•  Analysis  of  the  wing-leading  edge,  wing-trailing  edge,  and 
live  fire  test  data  for  the  center  wing  dry  bay  fire  confirmed 
the  vulnerability  of  the  KC-46A  to  dry  bay  fires.  Subsequent 


analysis  will  follow  completion  of  the  Fuselage  Dry  Bay  Fire 
Vulnerability  and  Wing  Dry  Bay  Fire  Sustainment  test  series  in 
FY15  to  determine  if  sufficient  data  were  obtained  to  quantify 
the  most  significant  dry  bay  fire  ignition  and  sustainment 
factors  affecting  overall  KC-46A  survivability. 

•  Live  fire  test  data  identified  the  KC-46  threat-induced 
structural  limitations  as  a  function  of  several  engagement 
conditions.  Post-test  data  analysis  is  ongoing  to:  (1)  improve 
modeling  and  simulation  accuracy,  (2)  assess  flight 
load-carrying  capability  for  engagement  conditions  beyond  the 
test  design  space,  and  (3)  assess  structural  failure  limits  as  a 
function  of  dynamic  loads. 

•  Preliminary  analysis  of  the  KC-46  armor  performance 
demonstrated  the  expected  armor  effectiveness  against  a 
specification  threat  with  80  percent  confidence.  Test  data  were 
also  collected  to  assess  the  performance  of  the  installed  armor 
against  the  specification  threat  and  two  other  operationally 
representative  threats.  Evaluation  of  the  effects  of  these  data 
on  the  overall  crew  protection  assessment  is  ongoing  and  will 
address  a  range  of  engagement  conditions. 

•  The  KC-46  EMP  design  margin  was  based  on  Military 
Standard  (MIL-STD)-2169.  After  the  fixed-price  contract 
was  awarded,  a  new  MIL-STD-3023  was  released  that  called 
for  tanker  aircraft  to  meet  a  20  dB  EMP  design  margin  versus 
a  6  dB  EMP  design  margin.  The  actual  KC-46  EMP  design 
margin  should  be  determined. 

•  The  TEMP  and  other  test  documents  do  not  address  detailed 
Information  Assurance  (IA)  protect,  detect,  react,  and  restore 
requirements.  The  program  has  begun  to  address  these 
shortfalls  by  planning  additional  testing  and  crew  IA  training 
through  the  IA  Working  Group. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 
addressed  one  of  the  FY12  recommendations  to  incorporate 
realistic  assumptions  in  test  plans;  however,  additional  work  is 
still  needed.  The  Air  Force  still  needs  to  address  the  remaining 
FY12  and  FY13  recommendations  to: 

1 .  Submit  a  TEMP  with  a  realistic  schedule  mitigating  the 
above  mentioned  shortfalls. 

2.  Provide  an  approach  to  correct  the  ALR-69A  RWR 
shortfalls  prior  to  integration  on  the  KC-46A. 

3.  Plan  to  begin  IOT&E  at  least  12  months  later  than  the 
current  TEMP  indicates  to  allow  for  completion  of 
developmental  test  and  initial  training. 

4.  Provide  a  comprehensive  aerial-refueling  certification  plan 
for  the  KC-46A  including  all  EMD  Phase  1  and  2  receivers. 

5.  Plan  testing  against  realistic  cybersecurity  threats  to  identify 
vulnerabilities  for  correction.  In  addition,  plan  follow-on 
penetration  testing  to  assess  IA  performance  in  terms  of 
protect,  detect,  react,  and  restore  functions. 

•  FY 14  Recommendation. 

1 .  The  Air  Force  should  conduct  EMP  testing  to  assess  the 
actual  EMP  design  margin  of  the  KC-46. 
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Massive  Ordnance  Penetrator  (MOP) 


Executive  Summary 

•  In  October  2014,  the  Air  Force  successfully  completed 
one  weapon  drop  from  the  B-2  aircraft  on  a  representative 
target.  The  test,  conducted  at  the  White  Sands  Missile 
Range,  New  Mexico,  demonstrated  weapon  behavior  after 
planned  enhancements  were  incorporated. 

•  DOT&E  intends  to  publish  a  classified  Early  Fielding  Report 
in  early  2015  to  summarize  the  FY15  testing  of  the  Enhanced 
Threat  Reduction  (ETR)  Phase  2  effort. 

System 

•  The  GBU-57  Massive  Ordnance  Penetrator  (MOP)  is  a 
large,  GPS-guided,  penetrating  weapon  with  the  ability  to 
attack  deeply-buried  and  hardened  bunkers  and  tunnels.  The 
warhead  case  is  made  from  a  special  high-performance  steel 
alloy  and  its  design  allows  for  a  large  explosive  payload 
while  maintaining  the  integrity  of  the  penetrator  case  during 
impact. 

•  The  B-2  Spirit  is  the  only  aircraft  in  the  Air  Force 
programmed  to  employ  the  MOP. 

•  The  GBU-57  warhead  is  more  powerful  than  its 
predecessors,  the  BLU-109  and  GBU-28. 

•  The  MOP  is  an  Air  Force-led,  Quick  Reaction  Capability  that 
is  on  DOT&E  oversight,  as  well  as  a  Secretary  of  Defense 
special  interest  effort. 


Mission 

Combatant  Commanders  use  MOP  to  conduct  pre-planned,  day 
or  night  attacks  against  defended  point  targets  vulnerable  to  blast 
and  fragmentation  effects  and  requiring  significant  penetration, 
such  as  hardened  and  deeply-buried  facilities. 

Major  Contractor 

The  Boeing  Company,  Defense,  Space  &  Security  -  St.  Louis, 
Missouri 


Activity 

•  Prior  to  the  live-flight  missions,  the  Air  Force  conducted 
one  static  test  and  flew  one  captive-carry  test  in  August  and 
September  2014,  to  validate  the  hardware  and  software 
changes  implemented  in  the  MOP. 

•  In  October  20 1 4,  the  Air  Force  executed  one  weapon 
drop  at  White  Sands  Missile  Range,  New  Mexico,  on  a 
representative  target.  This  testing  was  to  evaluate  the  effect 
of  the  ETR  Phase  2  modifications  to  the  weapon  system 
performance.  An  Air  Force  B-2  aircraft  flew  one  mission 
to  complete  the  drop.  Two  free-flight  missions,  one  inert 
and  one  live  warhead,  remain  to  complete  ETR  Phase  2 
testing. 


Assessment 

•  Both  the  static  and  captive-carry  tests  were  successful  and 
allowed  the  program  to  proceed  to  live  weapons  employment. 

•  The  Air  Force  and  DOT&E  are  currently  assessing  the  results 
from  the  MOP  ETR  Phase  2  effort,  and  will  report  them  in  a 
classified  DOT&E  Early  Fielding  Report  in  early  2015. 

Recommendations 

•  Status  of  Previous  Recommendations.  While  there  were 
no  previous  Annual  Report  recommendations  for  this 
program,  the  Air  Force  addressed  all  recommendations  in  the 
September  2013  Early  Fielding  Report. 

•  FY 14  Recommendations.  None. 
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Miniature  Air-Launched  Decoy  (MALD)  and 
MALD-Jammer  (MALD-J) 


Executive  Summary 

•  The  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  completed  full  mission-level  simulation  testing  for 
the  Miniature  Air-Launched  Decoy  -  Jammer  (MALD-J)  in 
February  2014. 

•  In  FY14,  the  Air  Force  launched  3  MALD-J  and  2  MALD 
vehicles  in  operational  environments. 

•  Preliminary  analysis  of  the  Force  Development  Evaluation 
(FDE)  indicates  persistent  problems  exist  with  MALD-J 
navigational  accuracy.  The  MALD-J  Program  Office  has 
incorporated  software  upgrades  to  improve  navigational 
accuracy,  but  the  changes  thus  far  have  focused  on  improving 
the  missile’s  altitude  hold  capability  and  have  improved 
navigation  accuracy  only  slightly. 

•  The  28th  Test  and  Evaluation  Squadron  is  currently  executing 
an  FDE,  in  conjunction  with  a  MALD-J  Reliability  Assessment 
Program  mission,  to  evaluate  MALD-J’s  improvement  to 
navigational  accuracy  due  to  software  upgrades. 

•  Preliminary  results  of  MALD  and  MALD-J  IOT&E  indicate 
the  Air  Force’s  corrective  actions  on  MALD-J  have  improved 
the  materiel  reliability. 

System 

•  The  MALD  is  a  small,  low-cost,  expendable,  air-launched 
vehicle  that  replicates  how  fighter,  attack,  and  bomber  aircraft 
appear  to  enemy  radar  operators. 

•  The  Air  Force  designed  the  MALD-J  as  an  expendable,  close-in 
jammer  to  degrade  and  deny  an  early  warning  or  acquisition 
radar’s  ability  to  establish  a  track  on  strike  aircraft  while 
maintaining  the  ability  to  fulfill  the  MALD  decoy  mission. 

•  In  FY12,  the  Program  Office  converted  the  MALD 
procurement  line  to  MALD-J.  The  Air  Force  will  no  longer 
procure  any  MALDs  without  the  jammer. 


•  The  F-16C/D  and  B-52  are  the  lead  aircraft  to  employ  MALD 
and  MALD-J. 

Mission 

Combatant  Commanders  will  use  units  equipped  with: 

•  MALD  and  MALD-J  to  improve  battlespace  access  for 
airborne  strike  forces  by  deceiving,  distracting,  or  saturating 
enemy  radar  operators  and  Integrated  Air  Defense  Systems. 

•  MALD  to  allow  an  airborne  strike  force  to  accomplish  its 
mission  by  deceiving  enemy  radars  and  forcing  air  defense 
systems  to  treat  MALD  as  a  viable  target. 

•  MALD-J  to  allow  an  airborne  strike  force  to  accomplish  its 
mission  by  jamming  enemy  radars  and  air  defense  systems  to 
degrade  or  deny  detection  of  friendly  aircraft  or  munitions. 

Major  Contractor 

Raytheon  Missile  Systems  -  Tucson,  Arizona 


Activity 

•  AFOTEC  completed  full  mission-level  simulation  testing  in 
February  2014  in  accordance  with  the  DOT&E-approved  test 
plan. 

•  In  April  2014,  the  Program  Office  completed  a  MALD-J 
Reliability  Assessment  Program  that  launched  a  single  MALD 
vehicle  to  test  and  verify  the  software  updates  for  navigation 
accuracy. 

•  The  28th  Test  and  Evaluation  Squadron  is  currently 
executing  an  FDE,  in  conjunction  with  a  MALD-J  Reliability 
Assessment  Program  mission,  to  evaluate  MALD-J’s 


improvement  to  navigational  accuracy  due  to  software 
upgrades.  The  test  squadron  has  accomplished  all  six  of  the 
planned  MALD-J  missile  launches. 

•  AFOTEC  published  an  operational  test  report  on  MALD-J 
in  February  2014.  DOT&E  is  waiting  to  publish  its  IOT&E 
report  until  after  MALD/MALD-J  navigational  improvements 
have  been  tested;  the  report  is  expected  to  be  released  in 
2QFY15. 

•  In  FY14,  the  Air  Force  launched  3  MALD-J  and  2  MALD 
vehicles  in  operational  environments. 
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Assessment 

•  DOT&E’s  April  2011  IOT&E  report  evaluated  MALD  as 
operationally  effective  for  combat,  blit  not  operationally 
suitable  due  to  poor  materiel  reliability.  AFOTEC’s  MALD 
IOT&E  report,  published  in  January  2012,  determined  MALD 
to  be  operationally  effective  and  suitable  with  identified 
mission  planning  and  reliability  shortfalls.  DOT&E  is  using 

a  combination  of  MALD  and  MALD-J  IOT&E  data  to 
evaluate  whether  the  Air  Force  has  mitigated  vehicle  reliability 
problems.  Since  no  failures  in  the  MALD-J  payload  to  date 
have  occurred,  and  the  missile  bodies,  flight  control  surfaces, 
and  navigational  systems  are  identical  to  MALD,  combining 
these  data  is  appropriate. 

•  Corrective  actions  appear  to  have  improved  the  poor  materiel 
reliability  in  the  intended  operational  environment. 

•  Preliminary  analysis  of  the  FDE  indicates  persistent  problems 
exist  with  MALD-J  navigational  accuracy.  Although  some 
corrections  have  demonstrated  minor  navigational  accuracy 
improvements  to  date,  overall  navigational  accuracy  problems 
in  most  operational  environments  persist. 

•  The  Air  Force  will  fully  implement  and  test  navigational 
accuracy  upgrades  in  FOT&E  in  FY15. 


•  Full  mission-level  planning  and  testing  events  for  the 
MALD-J  program  indicate  the  time  needed  to  plan  a  full  load 
of  MALD-J  vehicles  is  excessive.  Typically,  planning  time 
required  for  an  F-16  is  just  over  2  hours  for  a  full  mission,  but 
typical  planning  time  for  a  B-52  can  reach  over  12  hours. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 
satisfactorily  addressed  one  of  three  FY13  recommendations 
by  addressing  the  validation  and  accreditation  issues  involved 
with  the  Digital  Integrated  Air  Defense  model.  The  Air  Force 
is  currently  testing  to  address  the  remaining  recommendations 
to  improve  navigational  accuracy  in  operational  environments 
and  mission-planning  capabilities  for  the  MALD-J  program  to 
reduce  the  time  needed  to  plan  a  full  load  of  MALD-J  vehicles. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1 .  Complete  FDE  testing  to  evaluate  the  MALD-J  navigation 
system  improvements  due  to  software  upgrades. 

2.  Continue  to  develop  a  plan  to  fully  test  navigation  system 
upgrades  in  FOT&E. 
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MQ-9  Reaper  Armed  Unmanned  Aircraft  System  (UAS) 


Executive  Summary 

•  The  MQ-9  Program  of  Record  continues  to  face  challenges 
with  evolving  content,  prioritizing  and  maturing  system 
software,  and  developing  and  delivering  technical  order  data 
to  meet  development  and  fielding  timelines  for  the  MQ-9 
Increment  One  Program  of  Record.  The  Air  Force  will 
complete  MQ-9  remotely-piloted  aircraft  (RPA)  production 
under  Low-Rate  Initial  Production;  there  will  be  no  Full-Rate 
Production  decision. 

•  The  Air  Force  completed  a  Force  Development  Evaluation 
(FDE)  of  the  Block  1  RPA  configured  with  Operational  Flight 
Program  (OFP)  904.2  in  December  2013.  This  increment 

of  operational  testing  assessed  improvements  to  optical 
and  infrared  sensor  target  location  accuracy;  established  a 
baseline  measurement  of  radar  target  location  accuracy  and 
ground-moving  target  indicator  detection  capability;  and 
evaluated  system  user  interface  improvements. 

•  Block  5  RPA  developmental  testing  revealed  aircraft 
overheating  problems  that  precluded  the  completion  of 
operationally  representative  hot-weather  ground  testing.  At 
the  end  of  FY14,  the  Air  Force  was  pursuing  corrective  actions 
to  include  system  redesign  and  additional  developmental 
testing  in  FY15  in  order  to  meet  Air  Force  global  operating 
environment  requirements. 

•  The  final  configuration  of  the  MQ-9  Increment  One  Unmanned 
Aircraft  System  (UAS)  continued  to  evolve  in  FY14.  The 
Air  Force  proposed  further  changes  and  enhancements  to 

the  MQ-9  UAS.  These  changes  entail  additional  hardware, 
software,  and  enhanced  capabilities  beyond  those  addressed  in 
the  current  MQ-9  Test  and  Evaluation  Master  Plan  (TEMP). 
Changes  to  the  Increment  One  UAS  will  be  supported  by  a 
new  MQ-9  acquisition  strategy  and  new  developmental  and 
operational  test  and  evaluation  construct.  A  new  TEMP  is 
required  to  articulate  the  developmental  and  operational  test 
construct  and  scope  of  resources  necessary  to  support  the 
testing  of  the  MQ-9  UAS  content  proposed  under  the  new  Air 
Force  acquisition  strategy. 

System 

•  The  MQ-9  Reaper  UAS  is  a  remotely-piloted,  armed,  air 
vehicle  that  uses  optical,  infrared,  and  radar  sensors  to  locate, 
identify,  target,  and  attack  ground  targets. 

-  The  MQ-9  RPA  is  a  medium-sized  aircraft  that  has 
an  operating  ceiling  up  to  50,000  feet,  an  internal 
sensor  payload  of  800  pounds,  an  external  payload 
of  3,000  pounds,  and  an  endurance  of  approximately 
14  hours. 

-  The  Ground  Control  Station  (GCS)  commands  the  MQ-9 
RPA  for  launch,  recovery,  and  mission  control  of  sensors 


and  weapons.  C-band  line-of-sight  datalinks  are  used  for 
RPA  launch  and  recovery  operations,  and  Ku-band  satellite 
links  are  used  for  RPA  mission  control. 

•  The  MQ-9  RPA  carries  AGM-1 14,  HELLFIRE  II  anti-armor 
precision  laser-guided  missiles  and  GBU-12,  500-pound  laser 
guided  bombs. 

•  The  Air  Force  is  using  an  evolutionary  acquisition  approach 
for  meeting  Increment  One  Capability  Production  Document 
requirements,  with  Block  1  and  Block  5  RPAs  and  Block  15 
and  Block  30  GCSs. 

•  The  Air  Force  is  currently  fielding  the  Block  1  RPA  and  the 
Block  15  GCS. 

•  The  Air  Force  designed  the  Block  5  RPA  to  incorporate 
improved  main  landing  gear,  an  upgraded  electrical  system 
with  more  power,  an  additional  ARC-210  radio,  encrypted 
datalinks,  a  redesigned  avionics  bay  and  digital  electronic 
engine  control  system,  the  BRU-71  bomb  rack,  high-definition 
video,  and  upgraded  software  to  allow  the  two-person  aircrew 
to  operate  all  onboard  systems. 

Mission 

•  Combatant  Commanders  use  the  MQ-9  onboard  sensors  and 
weapons  to  conduct  armed  reconnaissance  and  pre-planned 
strikes.  Units  equipped  with  MQ-9s  can  find,  fix,  track,  target, 
engage,  and  assess  critical  emerging  targets  (both  moving  and 
stationary). 

•  MQ-9  units  can  also  conduct  aerial  intelligence  gathering, 
reconnaissance,  surveillance,  and  target  acquisition  for  other 
airborne  platforms. 

Major  Contractor 

General  Atomics  Aeronautical  Systems  Inc.  -  San  Diego, 

California 
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Activity 

•  The  Air  Force  conducted  all  MQ-9  testing  in  accordance  with 
the  DOT&E-approved  TEMP  and  test  plan. 

•  Air  Combat  Command  completed  an  FDE  of  the  Block  1  RPA 
configured  with  OFP  904.2  in  December  2013.  This  increment 
of  operational  testing  assessed  improvements  to  optical  and 
infrared  sensor  target  location  accuracy;  established  a  baseline 
measurement  of  radar  target  location  accuracy  and  ground 
moving  target  indicator  (GMTI)  detection  capability;  and 
evaluated  system  user  interface  improvements. 

•  In  June  and  July  of  2014,  the  Air  Force  conducted  MQ-9 
Block  5  RPA  developmental  testing  to  include  an  Integrated 
System  Evaluation  of  hot-weather  capabilities.  Block  5  RPA 
design  shortfalls  resulted  in  aircraft  overheating  on  the  ground 
that  precluded  the  Air  Force  from  successfully  completing  this 
testing. 

•  In  July  2014,  the  Air  Force  92d  Information  Operations 
Squadron  and  the  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  conducted  a  cybersecurity  Cooperative 
Vulnerability  Assessment  of  the  Block  5  RPA  and  Block  30 
GCS  to  support  the  planned  FY15  FOT&E. 

•  Late  in  F Y 1 4,  Air  Combat  Command  began  a  Tactics 
Investigation  to  explore  the  potential  for  limited  envelope 
employment  of  GBU-38  500-pound  Joint  Direct  Attack 
Munition  (JDAM)  weapons  from  Block  1  RPAs  configured 
with  OFP  904.2.  AFOTEC  will  fully  test  JDAM  capabilities 
during  planned  FY15  FOT&E  of  the  MQ-9  Block  5  RPA  and 
Block  30  GCS  configured  with  OFP  904.6. 

•  The  final  configuration  of  the  MQ-9  Increment  One  UAS 
continued  to  evolve  throughout  FY14.  As  of  the  end  of 
FY14,  the  Air  Force  indicated  it  intends  to  incorporate  an 
improved  Multi-Spectral  Targeting  System-B  (MTS-B) 
electro-optical/infrared  sensor,  additional  weapons,  avionics 
hardware,  and  further  system  software  revisions  into  the 
Increment  One  Program  of  Record  capabilities. 

•  At  the  end  of  FY14,  the  MQ-9  System  Program  Office  was 
exploring  new  acquisition  strategy  approaches  to  react  to 
changing  content  desired  by  the  Air  Force  and  to  deliver 
desired  capabilities  to  the  users. 

Assessment 

•  The  MQ-9  program  continues  to  face  systemic  challenges 
in  prioritizing  and  maturing  software  OFPs  and  developing 
technical  order  data  to  meet  development  and  fielding 
timelines  for  the  MQ-9  Increment  One  Program  of  Record. 

The  Air  Force  has  elected  not  to  conduct  a  Full-Rate 
Production  decision  for  the  Block  5  RPA,  and  the  MQ-9 
system  will  complete  delivery  of  all  planned  RPAs  under 
Low-Rate  Initial  Production.  FOT&E  of  the  Increment  One 
UAS  configuration,  originally  planned  for  2013,  is  projected 
to  begin  in  late  FY15. 

•  DOT&E  assesses  that  results  of  the  MQ-9  904.2  software 
upgrade  FDE  demonstrate: 

-  The  MTS-B  e  lec  tro-  opt  ical/in  fra  red  targeting  accuracy 
is  improved  compared  to  that  of  the  legacy  OFP 
configuration,  and  supports  employment  of  legacy 


AGM-114  HELLFIRE  missiles  and  GBU-12  500-pound, 
laser-guided  bombs. 

-  The  Lynx  synthetic  aperture  radar  (SAR)  targeting 
accuracy  supports  target  designation  and  employment 
of  AGM-1 14  missiles  and  GBU-12  laser-guided 
bombs.  Lynx  SAR  testing  results  further  demonstrated 
system  target  location  accuracies  sufficient  to  support 
JDAM  employment.  However,  JDAM  testing  was  not 
accomplished  as  part  of  the  OFP  904.2  FDE  due  to  JDAM 
launch  acceptability  region  software  deficiencies.  The 
Air  Force  intends  to  correct  JDAM  launch  acceptability 
region  shortfalls  in  future  MQ-9  OFP  904.6. 

-  The  Lynx  SAR  GMTI  demonstrated  the  ability  to  detect 
ground-moving  targets;  however,  cross-cueing  capability 
from  the  Lynx  SAR  to  the  MTS-B  was  poor. 

-  Neither  the  RPA  nor  GCS  met  the  Air  Force  Mean  Time 
Between  Critical  Failure  or  Mean  Repair  Time  threshold 
requirements  during  the  FDE  period.  However,  16  of  16 
RPA  failures  and  17  of  20  GCS  failures  were  hardware 
failures  not  related  to  the  OFP  upgrade.  The  only  software 
anomalies  observed  were  confined  to  the  GCS  where  two 
system  software  resets  and  one  software  reload  occurred. 

•  Developmental  testing  of  the  MQ-9  Block  5  RPA  revealed 
system  deficiencies  that  precluded  the  successful  completion 
of  planned  FY14  hot  weather  testing.  Testers  were  unable 
to  execute  planned  sorties  in  an  operationally  representative 
hot-weather  environment  due  to  the  Block  5  RPA  system 
overheating  during  ground  operations  prior  to  take-off. 

-  Based  on  the  performance  shortfalls  encountered,  the 
Air  Force  is  considering  redesigned  hardware,  additional 
ground  equipment,  OFP  software  changes,  and  technical 
order  changes  to  resolve  the  overheating  problems. 

-  However,  at  the  end  of  FY14,  a  final  solution  had  not  been 
identified  and  additional  developmental  testing  remained 
to  be  accomplished.  The  Air  Force  expects  to  resolve  this 
problem  prior  to  the  start  of  planned  FY15  FOT&E. 

•  Preliminary  observations  of  Air  Combat  Command’s 
Tactics  Investigation  exploring  limited  envelope  operational 
employment  of  GBU-38  JDAMs  indicate  that  although 
weapons  can  be  employed  from  MQ-9  Block  1  RPAs 
configured  with  OFP  904.2,  current  displayed  launch 
acceptability  region  shortfalls  do  not  yet  meet  Air  Force 
operational  needs.  JDAM  employment  remains  to  be  fully 
evaluated  during  the  FY15  FOT&E  in  conjunction  with 
OFP  904.6  testing. 

•  The  DOT&E-approved  TEMP  identifies  OT&E  requirements 
for  the  currently-defined  final  configuration  of  the  MQ-9 
Increment  One  UAS.  Should  the  configuration  change  from 
that  identified  in  the  TEMP  (Block  5  RPA,  Block  30  GCS,  and 
OFP  904.6),  an  updated  TEMP  will  be  required  to  address  new 
Increment  One  UAS  configurations,  content,  and  associated 
T&E  efforts.  Regardless  of  any  potential  changes,  DOT&E 
will  evaluate  the  Block  5  RPA,  Block  30  GCS,  and  OFP  904.6 
in  conjunction  with  the  planned  AFOTEC  FOT&E  in  FY15. 
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Recommendations 

•  Status  of  Previous  Recommendations.  In  FY 14,  the 

Air  Force  made  progress  towards  satisfying  previous  years’ 
recommendations  to  complete  the  development  of  the 
Increment  One  UAS  hardware  and  software  cybersecurity 
vulnerability  testing. 

•  FY  14  Recommendations.  The  Air  Force  should: 

1.  Complete  the  development  of  the  Increment  One  UAS 
hardware  and  software  to  support  FOT&E  of  the 
Increment  One  system. 

2.  Improve  the  Lynx  SAR  GMTI  cross-cueing  capability  to 
enable  effective  hunter/killer  operations  against  moving 
targets. 


3.  Resolve  the  Block  5  RPA  hot-weather  operating  shortfalls 
prior  to  the  start  of  AFOTEC’s  planned  FOT&E  in  FY15. 

4.  Complete  the  development  of  the  GBU-38  JDAM 
capability  for  MQ-9  and  fully  test  it  during  AFOTEC’s 
FOT&E  in  FY15. 

5.  Update  the  MQ-9  Increment  One  UAS  TEMP  to  accurately 
reflect  the  content  being  pursued  under  the  evolving 
program  of  record. 
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QF-16  Full-Scale  Aerial  Target  (FSAT) 


Executive  Summary 

•  The  Air  Force  executed  99  QF- 1 6  developmental  and 
operational  test  sorties  in  FY13  and  14,  completing  IOT&E  in 
September  2014. 

•  The  QF-16  demonstrated  progress  toward  mission 
effectiveness  and  suitability  as  a  fourth-generation  Full-Scale 
Aerial  Target  (FSAT)  to  support  test,  evaluation,  and  training 
of  U.S.  weapon  systems  at  the  Eglin  Gulf  Test  and  Training 
Range,  Florida,  and  White  Sands  Missile  Range,  New  Mexico. 

•  The  Air  Force  should  complete  QF-16  radar  cross  section 
measurements  and  ensure  procurement  funding  provides  at 
least  25  FSAT  targets  per  year  beginning  in  FY16  to  meet 
Service-coordinated  aerial  target  requirements,  in  compliance 
with  Resource  Management  Decision  700. 

•  The  Air  Force  should  provide  plans  for  Phase  II  of  the  Air 
Superiority  Target  program  to  address  shortfalls  in  testing 
against  fifth-generation  airborne  threats. 

System 

•  The  QF-16  is  the  latest  FSAT  designed  to  test  and  evaluate 
U.S.  weapon  systems  and  assist  in  developing  tactics, 
techniques,  and  procedures  to  counter  fighter-size  airborne 
threats.  The  DOD  is  replacing  the  current  FSAT,  the  QF-4, 
due  to  its  increasing  dissimilarity  from  current  and  projected 
air-superiority  threats,  declining  supportability,  and  depletion 
of  suitable  F-4  airframes. 

•  The  QF-16  system  is  composed  of  regenerated  F-16  Block  15, 
25,  and  30  aircraft  equipped  with  Drone-Peculiar  Equipment 
to  enable  remote  command  and  control,  missile  trajectory 
scoring,  and  safe  flight  termination.  Like  the  QF-4,  the  QF-16 
is  capable  of  manned  and  Not  Under  Live  Local  Operator 
(NULLO)  flight  operations.  It  will  operate  from  Tyndall 
AFB,  Florida,  using  the  Gulf  Range  Drone  Control  System, 
and  Holloman  AFB,  New  Mexico,  using  the  White  Sands 
Integrated  Target  System  located  at  White  Sands  Missile 
Range,  New  Mexico. 


•  The  QF-16  retains  F-16  flight  performance  characteristics 
and  payload  capabilities  including  supersonic,  after-burning 
engines,  high-G  maneuvering,  complex  electronic  attack,  and 
expendable  countermeasures. 

Mission 

The  DOD  uses  FSATs  to: 

•  Provide  threat-representative  presentations  for  developmental 
and  operational  test  and  evaluation  for  U.S.  weapon  systems, 
as  mandated  by  Title  10  U.S.  Code,  Section  2366 

•  Continuously  evaluate  fielded  air-to-air  missile  capabilities 
while  providing  live  missile  training  for  combat  air  crews 
through  Air  Force  and  Navy  Weapon  Systems  Evaluation 
Programs 

Major  Contractor 

The  Boeing  Company  -  St.  Louis,  Missouri 


Activity 

•  QF-16  completed  its  last  IOT&E  flight  on  September  5,  2014. 
In  total,  QF-16  flew  99  developmental  and  operational  test 
sorties,  19  of  which  were  integrated  test,  and  4  dedicated 
operational  test  sorties  (2  manned  and  2  NULLO). 

•  The  QF-16  program  met  Milestone  C  in  October  2013  after 
nine  months  of  developmental  and  integrated  testing  with  the 
Eglin  Gulf  Test  and  Training  Range  at  Tyndall  AFB  and  Cecil 
Field  in  Florida. 

•  The  QF-16  program  relocated  to  Holloman  AFB,  New 
Mexico,  to  conduct  developmental,  integrated,  and  operational 


testing,  including  integration  with  the  White  Sands  Integrated 
Target  Control  System. 

•  On  May  8,  2014,  the  Program  Executive  Officer  convened 
the  Operational  Test  Readiness  Review  and  certified  QF-16  to 
begin  IOT&E. 

•  QF-16  completed  its  first  of  two  operational  test  flights  in  the 
NULLO  configuration  on  June  25,  2014,  at  Holloman  AFB. 
Mission  events  included  testing  of  the  Vector  Scoring  System 
utilizing  the  Center  for  Countermeasures  at  White  Sands 
Missile  Range  and  two  surface-to-air  missiles. 
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•  The  final  operational  test  NULLO  flight  occurred  on 
September  5,  2014,  in  conjunction  with  an  AIM-9X  Block  II 
operational  test  live  missile  shot,  which  provided  a  realistic 
test  of  the  QF-16  Vector  Scoring  System. 

•  The  program  scheduled  the  QF-16  Full-Rate  Production 
decision  for  2QFY 1 5  with  first  delivery  to  the  82d  Aerial 
Targets  Squadron,  Tyndall  AFB,  at  the  end  of  February  2015. 
The  Air  Force  plans  to  achieve  Initial  Operational  Capability  at 
Tyndall  AFB  in  1QFY16. 

•  The  Air  Force  conducted  the  IOT&E  in  accordance  with  the 
DOT&E-approved  Test  and  Evaluation  Master  Plan. 

Assessment 

•  QF-16  has  demonstrated  progress  to  provide  an  FSAT 
representing  fourth-generation  airborne  threats  for  U.S. 
weapon  systems  testing.  Analysis  and  reporting  of  IOT&E 
is  incomplete  but  will  conclude  in  2QFY15;  in  particular, 
fixes  to  the  Vector  Scoring  System  that  were  identified  in 
developmental  testing  remain  to  be  verified. 

•  QF-16  has  demonstrated  progress  in  suitability  metrics  under 
the  current  Air  Force  concept  of  operations  that  generates 

a  primary  and  backup  FSAT  for  each  chargeable  scheduled 
sortie  to  achieve  a  95  percent  threshold  supportability  rating 
(defined  as  the  ability  to  successfully  launch  in  support  of 
a  scheduled  test  event).  The  Drone-Peculiar  Equipment 
operating  time  totaled  776.6  hours  with  25  failures,  resulting 
in  a  3 1.1 -hour  Mean  Time  Between  Failure,  below  the 
Capabilities  Development  Document  threshold  requirement 
of  45  hours.  The  Army  Materiel  Systems  Analysis  Activity 
Maturity  Project  Model  predicts  QF-16  will  reach  31.2  hours 
by  Full  Operational  Capability  in  FY17. 

•  The  Air  Force  did  not  require  QF-16  to  represent 
fifth-generation  airborne  threat  systems  (including  radio 


frequency  low  observability  characteristics,  internally-carried 
advanced  electronic  attack,  and  low  probability  of  intercept 
sensors).  DOT&E  continues  to  emphasize  existing  aerial 
targets,  including  the  QF-16,  are  insufficient  for  adequate 
operational  testing  of  U.S.  weapon  systems. 

-  In  the  Air  Superiority  Target  Phase  1  Analysis  of 
Alternatives  Final  Report  (March  15,  2007),  the  Air  Force 
recommended  further  study  to  produce  user  consensus 

on  critical  characteristics  of  future  aerial  targets  and 
to  determine  capabilities  and  shortfalls  in  existing  test 
resources. 

-  Multiple  stakeholders  within  Congress,  OSD,  the  Air 
Force,  and  the  Navy,  support  the  requirement  for  a 
fifth-generation  FSAT. 

Recommendations 

•  Status  of  Previous  Recommendations.  This  is  the  first  annual 
report  for  this  program. 

•  FY 14  Recommendations.  The  Air  Force  should: 

1.  Complete  radar  cross  section  measurements  for  QF-16  to 
ensure  current  and  future  U.S.  weapon  systems  programs 
have  precise,  reliable  data  on  system  performance  against 
measured,  low-observable  target  presentations. 

2.  Ensure  QF-16  procurement  funding  continues  to  comply 
with  Resource  Memorandum  Decision  700-mandated  levels 
of  25  aircraft  per  year  beginning  in  FY16,  in  order  to  meet 
Service-coordinated  and  approved  test  and  training  resource 
requirements. 

3.  Complete  the  user  requirements  and  current  capabilities 
studies  and  provide  plans  for  Phase  II  of  the  Air  Superiority 
Target  program  to  address  test  and  evaluation  shortfalls  for 
U.S.  weapon  systems  with  respect  to  threat-representative, 
fifth-generation  FSATs. 
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RQ-4B  Global  Hawk  High-Altitude  Long-Endurance 
Unmanned  Aerial  System  (UAS) 


Executive  Summary 

•  The  2015  Presidential  Budget  fully  funded  the  Global  Hawk 
program,  resolving  several  years  of  programmatic  uncertainty. 
As  a  result,  the  Air  Force  has  taken  delivery  of  1 8  of  2 1  RQ-4B 
Block  30  and  11  RQ-4B  Block  40  air  vehicles.  Additionally, 
the  program  has  delivered  all  Mission  Control  Elements  and 
Launch  and  Recovery  Elements. 

•  The  Air  Force  plans  to  re-establish  and  formalize  a  combined 
RQ-4B  Global  Hawk  Block  30/40  baseline  program.  In 
preparation  for  a  future  Milestone  C  decision  for  the  revised 
program,  the  Air  Force  is  working  to  define  baseline  “as  is” 
system  configurations,  define  future  operational  capability 
requirements,  develop  a  revised  acquisition  and  sustainment 
strategy,  and  produce  a  comprehensive  RQ-4B  Global  Hawk 
Test  and  Evaluation  Master  Plan  (TEMP)  to  test  and  evaluate 
the  correction  of  deficiencies  and  the  effectiveness  of  future 
enhancements.  In  addition,  the  Air  Force  is  developing  RQ-4B 
Block  30  operational  test  schedules  and  strategies  to  submit 

to  DOT&E  for  review  in  a  revised  program  TEMP  90  days 
after  the  Block  30/40  Capabilities  Production  Documents  are 
approved  by  the  Joint  Requirements  Oversight  Council. 

•  The  Air  Force  continued  developmental  testing  of  the  RQ-4B 
Block  40  system  in  FY14  in  preparation  for  the  planned 
IOT&E  in  FY15. 

•  The  program  continued  to  support  operational  employment 
with  two  Block  40  systems  in  the  U.S.  Central  Command 
(USCENTCOM)  area  of  operations  and  two  systems  to  the 
U.S.  Pacific  Command  (USPACOM)  area  of  operations.  The 
operational  capability  of  these  systems  is  limited,  but  supports 
the  immediate  requirements  of  the  operational  forces. 

•  The  Air  Force  is  currently  developing  follow-on  operational 
test  strategies  for  future  system  enhancements  for  FY15  and 
beyond. 

System 

•  The  RQ-4B  Global  Hawk  is  a  remotely-piloted,  high-altitude, 
long-endurance  airborne  Intelligence,  Surveillance,  and 
Reconnaissance  system  that  includes  the  Global  Hawk 
unmanned  air  vehicle,  various  intelligence  and  communications 
relay  mission  payloads,  and  supporting  command  and  control 
ground  stations. 

•  The  RQ-4B  Global  Hawk  Block  30  system  is  equipped  with 
a  multi-intelligence  payload  that  includes  both  the  Enhanced 
Integrated  Sensor  Suite  imagery  intelligence  payload  and 
the  Airborne  Signals  Intelligence  Payload  (ASIP)  signals 
intelligence  sensor. 

•  The  RQ-4B  Block  40  system  is  equipped  with  the 
Multi-Platform  Radar  Technology  Insertion  Program 


(MP-RT1P)  synthetic  aperture  radar  payload  designed  to 
simultaneously  collect  imagery  intelligence  on  stationary 
ground  targets  and  track  ground-moving  targets. 

Mission 

Commanders  use  RQ-4  Global  Hawk  reconnaissance  units  to 

provide  high-altitude,  long-endurance  intelligence  collection 

capabilities  to  support  theater  operations. 

•  Operators  collect  imagery  and  signals  intelligence  data  in 
order  to  support  ground  units  and  to  identify  intelligence 
essential  elements  of  information  for  theater  commanders. 
Units  equipped  with  RQ-4B  Global  Hawk  use  line-of-sight 
and  beyond  line-of-sight  satellite  datalinks  to  control  the 
Global  Hawk  system  and  transmit  collected  intelligence  data. 

•  Distributed  intelligence  processing,  exploitation,  and 
dissemination  systems  receive  intelligence  data  directly  from 
the  air  vehicle  or  from  the  Global  Hawk  ground  station  via 
intelligence  data  transmission  systems. 

•  Ground-based  intelligence  analysts  exploit  collected  imagery, 
ground-moving  target,  and  signals  information  to  provide 
intelligence  products  in  support  of  theater  operations. 

•  Global  Hawk  can  also  provide  imagery  intelligence  directly  to 
forward-based  personnel  through  direct  line-of-sight  datalink 
systems. 

Major  Contractor 

Northrop  Grumman  Aerospace  Systems,  Strike  and  Surveillance 

Systems  Division  -  San  Diego,  California 
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Activity 

•  The  2015  Presidential  Budget  fully  funded  the  Global  Hawk 
program,  resolving  several  years  of  programmatic  uncertainty. 
As  a  result,  the  Air  Force  has  taken  delivery  of  1 8  of  2 1  RQ- 
4B  Block  30,  and  11  RQ-4B  Block  40  air  vehicles. 
Additionally,  the  program  has  delivered  all  nine  Mission 
Control  Elements  and  all  10  Launch  and  Recovery  Elements. 

•  The  Air  Force  is  completing  the  requirements  to  achieve 
a  Milestone  C  decision  that  will  establish  and  formalize  a 
combined  and  coherent  Global  Hawk  Block  30/40  baseline 
program. 

Block  30 

•  The  Air  Force  is  continuing  to  acquire  and  pursue  upgrade 
programs  for  the  ASIP  sensor.  In  addition,  the  Air  Force 
planned  to  modify  some  RQ-4B  Block  30  ASIP  sensors  for 
transfer  and  deployment  on  the  U-2  Dragon  Lady  in  FY14. 

•  The  Air  Force  is  developing  future  RQ-4B  Block  30 
operational  test  schedules  and  plans  to  be  submitted  to 
DOT&E  for  review  in  a  revised  program  TEMP  90  days 
after  the  Block  30/40  Capabilities  Production  Documents 
are  approved  by  the  Joint  Requirements  Oversight  Council. 

•  No  RQ-4B  Block  30  operational  testing  was  conducted  in 
FY14. 

Block  40 

•  The  Air  Force  continued  developmental  testing  of  the 
RQ-4B  Block  40  system  in  FY14  in  preparation  for  the 
planned  IOT&E  in  FY15. 

•  The  program  continued  to  support  operational  employment, 
with  two  systems  in  the  USCENTCOM  area  of  operations 
and  two  systems  to  the  USPACOM  area  of  operations. 

The  operational  capability  of  these  systems  is  limited,  but 
supports  the  immediate  requirements  of  the  operational 
forces. 

•  The  Air  Force  is  currently  developing  follow-on  operational 
test  plans  for  future  system  enhancements  for  FY15  and 
beyond. 

•  The  Air  Force  completed  six  planned  interoperability 
test  flights  and  three  integrated  system  evaluation  test 
events  with  U.S.  Air  Force  Distributed  Ground  Stations 
(AF  DGSs)  to  demonstrate  system  maturity  and  improve 
the  exploitation  and  dissemination  of  RQ-4B  Block  40  data. 

•  The  Air  Force  also  implemented  a  number  of  radar  system 
and  image  processing  changes  intended  to  address  synthetic 
aperture  radar  image  quality  problems  observed  during  the 
FY13  RQ-4B  Block  40  Operational  Utility  Evaluation.  The 
impact  of  these  software  and  processing  upgrades  will  be 
demonstrated  and  fully  evaluated  during  RQ-4B  Block  40 
IOT&E  in  FY15. 

Assessment 

Previous  acquisition  strategies  have  resulted  in  Block  30  and 
Block  40  being  managed  as  separate  programs.  With  the  full 
funding  in  the  2015  Presidential  Budget,  the  Air  Force  has 
revised  its  acquisition  strategy  and  is  working  to  bring  all 
Global  Hawk  variants  into  one  cohesive  program.  Block  30  and 


Block  40  have  separate  operational  test  requirements,  and  the 
Air  Force  is  working  to  develop  a  TEMP  that  will  set  a  strategy 
for  Block  30,  Block  40,  and  any  enhancements  required  in 
the  future.  This  combined  approach  will  codify  the  varied 
operational  test  requirements  and  identify  test  agencies  and 
resources  required  to  complete  the  evaluations. 

Block  30 

•  Since  the  combined  RQ-4B  Block  30/ASIP  IOT&E 
in  2011,  the  Air  Force  has  corrected  most  RQ-4B 
air  vehicle  reliability  and  availability  problems  and 
implemented  a  limited  number  of  previously  planned 
system  improvements.  However,  due  to  programmatic 
issues  resulting  from  the  previous  DOD  decision  to  retire 
the  RQ-4B  fleet,  the  Air  Force  has  not  yet  conducted  a 
comprehensive  FOT&E  to  verify  correction  of  all  major 
IOT&E  deficiencies.  Currently  fielded  RQ-4B  Block  30 
systems  continue  to  operate  with  many  of  the  same 
operational  performance  and  signals  intelligence  mission 
deficiencies  identified  during  IOT&E. 

•  Preliminary  program  plans  for  the  re-constituted  RQ-4B 
Block  30  program  include  FOT&E  events  to  assess 
SIGINT  mission  performance  using  the  ASIP  sensor, 
previously  identified  ground  station,  air  vehicle,  and 
communication  system  operational  deficiencies,  as 
well  as  a  number  of  interoperability  and  cybersecurity 
shortfalls.  The  test  schedule  for  future  operational  test 
events  depends  on  the  resolution  of  current  program 
problems  leading  to  a  USD(AT&L)-approved  program 
schedule. 

Block  40 

•  The  Air  Force  continued  to  execute  planned 
developmental  testing  leading  to  IOT&E  in  FY15.  Initial 
results  from  key  developmental  test  integrated  system 
evaluation  events  with  AF  DGS  show  improved  maturity 
in  MP-RTIP  radar  system  stability  and  interoperability. 

•  The  Program  Office  believes  it  has  implemented  fixes  that 
will  improve  sensor  stability  and  overall  suitability  prior 
to  IOT&E.  Field  data  from  USCENTCOM  early  fielding 
appear  to  indicate  that  software  fixes  and  procedural 
workarounds  have  improved  suitability  in  comparison 

to  FY13  Operational  Utility  Evaluation  performance. 

All  sensor  modes  and  the  enduring  Tasking,  Collection, 
Processing,  Exploitation  and  Dissemination  system  will 
be  rigorously  tested  and  fully  characterized  during  the 
IOT&E  in  FY15. 

•  Completion  of  Block  40  developmental  test  events 
was  delayed  due  to  slower  than  expected  maturation 
of  AF  DGS  software  necessary  to  receive  and  exploit 
MP-RTIP  radar  data.  IOT&E  execution  was  delayed 
from  1QFY15  to  3QFY15  due  to  a  combination  of 

AF  DGS  software  delays  and  expected  adverse  weather 
conditions  at  Grand  Forks  AFB,  North  Dakota. 
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Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 
has  not  yet  completed  an  RQ-4B  Block  30  or  RQ-4B 
Block  40  TEMP  to  guide  developmental  and  operational 
testing  of  these  systems,  nor  has  it  proceeded  with  an  ASIP 
sensor  FOT&E  event  to  verify  correction  of  performance 
deficiencies  identified  during  IOT&E.  The  Air  Force  has 
implemented  some  corrective  actions  for  persistent  RQ-4B 
Block  40/MP-RTIP  sensor  stability  and  radar  image  quality 
problems  and  is  currently  evaluating  system  performance  in 
advance  of  IOT&E  in  FY15. 


•  FY 14  Recommendations.  The  Air  Force  should: 

1 .  Develop  an  RQ-4B  program  TEMP  to  guide  completion  of 
post-IOT&E  corrective  actions  and  to  define  operational 
test  requirements  for  future  Block  30  and  Block  40  system 
upgrades. 

2.  Complete  RQ-4B  Block  40  developmental  testing  to  verify 
MP-RTIP  performance  and  stability  prior  to  IOT&E  in  FY15. 

3.  Develop  a  plan  to  complete  the  FOT&E  for  Global  Hawk 
Block  30  signals  intelligence  mission  to  fully  characterize 
system  capability. 
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Small  Diameter  Bomb  (SDB)  II 


Executive  Summary 

•  The  Small  Diameter  Bomb  II  (SDB  II)  program  has 
completed  eight  Guided  Test  Vehicle  (GTV)  shots  and 
one  live  fire  test  shot  as  part  of  developmental  testing  in 
preparation  for  System  Verification  Review  and  Milestone  C 
decision,  which  is  expected  in  FY15.  An  updated  Test  and 
Evaluation  Master  Plan  is  being  finalized  in  support  of  a 
Milestone  C  decision. 

•  The  Integrated  Test  Team  completed  work  to  examine  test 
resource  and  planning  requirements  for  developmental, 
live  fire,  and  operational  testing,  resulting  in  an  adequate 
test  program  as  SDB  II  proceeds  through  Engineering  and 
Manufacturing  Development  (EMD). 

System 

•  The  SDB  11  is  a  250-pound,  air-launched,  precision-glide 
weapon  that  uses  deployable  wings  to  achieve  stand-off 
range.  F-15E  aircraft  employ  SDBs  from  the  BRU-61/A 
four-weapon  carriage  assembly. 

•  SDB  II  combines  Millimeter- Wave  radar,  infrared,  and 
laser-guidance  sensors  in  a  terminal  seeker,  in  addition  to 
a  GPS  and  Inertial  Navigation  System  to  achieve  precise 
guidance  accuracy  in  all  weather.  The  SDB  11  incorporates 
a  multi-function  (blast,  fragmentation,  and  shaped  charged 
jet)  warhead,  designed  to  defeat  armored  and  non-armored 
targets.  The  weapon  can  be  set  to  initiate  on  impact,  at  a 
preset  height  above  the  intended  target,  or  in  a  delayed  mode. 

•  SDB  II  provides  increased  weapons  load  per  aircraft 
compared  to  legacy  air-to-ground  munitions  employment 
against  offensive  counter-air,  strategic  attack,  interdiction, 
and  close-air  support  targets  in  adverse  weather. 


•  SDB  provides  reduced  collateral  damage  while  achieving  kills 
across  a  broad  range  of  target  sets  by  precise  accuracy,  small 
warhead  design,  and  focused  warhead  effects. 

Mission 

•  Combatant  Commanders  will  use  SDB  II  to  attack  stationary 
and  moving  targets  in  degraded  weather  conditions  at 
standoff  ranges.  There  are  three  principal  attack  modes: 
Normal  Attack  (NA),  Semi-Active  Laser  (SAL)  Attack,  and 
Coordinate  Attack  (CA).  SDB  II  can  also  be  used  against 
moving  or  stationary  targets  using  its  NA  (radar/infrared 
sensors)  or  SAL  modes,  and  fixed  targets  with  its  CA  mode. 

•  An  SDB  II-equipped  unit  or  Joint  Terminal  Attack  Controller 
will  use  a  weapon  datalink  network  to  provide  in-flight  target 
updates,  in-flight  retargeting,  weapon  in-flight  tracking,  and 
weapon  abort. 

Major  Contractor 

Raytheon  Missile  Systems  -  Tucson,  Arizona 


Activity 

•  Over  the  past  two  years,  SDB  11  has  flown  1 1  GTV  and  2  live 
fire  missions  as  part  of  developmental  testing. 

•  The  Program  Office  completed  11  rounds  of  seeker  Captive 
Flight  Test,  resulting  in  over  1,000  target  runs  in  a  wide 
variety  of  terrain  and  environmental  conditions  providing 
terabytes  of  seeker  performance  data  and  over  400  hours  of 
seeker  operation  without  a  single  failure. 

•  Nearly  2,000  hours  of  reliability  testing  have  been  completed, 
and  work  on  the  Integrated  Flight  System  has  been  done  with 
verification  and  validation  on  track  for  completion  prior  to 
Milestone  C. 

•  The  Integrated  Test  Team  fully  examined  test  resource  and 
planning  requirements  for  developmental,  live  fire,  and 
operational  testing,  resulting  in  an  adequate  test  program  as 
SDB  II  proceeds  through  EMD. 


•  The  program  is  finalizing  the  Test  and  Evaluation  Master  Plan 
in  support  of  Milestone  C. 

Assessment 

•  SDB  II  continues  to  progress  through  EMD  with  an  adequately 
resourced  test  program  and  no  major  programmatic  testing 
problems  outstanding. 

•  Eight  of  the  1 1  GTV  missions  and  1  of  2  live  fire  missions 
were  successful,  leaving  only  1  live  fire  mission  remaining 
to  meet  the  testing  requirements  to  pass  Milestone  C.  The 
failures  were  thoroughly  investigated  and  corrective  actions 
implemented  before  proceeding  with  additional  tests.  All 
corrective  actions  have  been  successful  to  date  in  preventing 
repeats  of  the  observed  failure  modes. 
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•  As  SDB  II  has  a  small  payload,  degradation  in  weapon 
accuracy  or  warhead  lethality  can  lead  to  a  drop  in  weapon 
effectiveness.  Seeker  performance  and  the  ability  to  properly 
assess  that  performance,  as  well  as  warhead  lethality,  are 
critical  to  program  success.  Flying  test  bed  seeker  results  have 
been  the  predominant  source  of  data  on  seeker  performance 
during  the  first  years  of  EMD.  Modeling  and  simulation 
will  provide  tools  to  interpret  that  data  and  evaluate  weapon 
performance  throughout  program  development.  Both  are 
critical  aspects  of  the  EMD  program. 


Recommendations 

•  Status  of  Previous  Recommendations.  The  Air  Force 
completed  all  previous  recommendations. 

•  FY 14  Recommendation. 

1.  The  SDB  II  Program  Office  should  pay  particular  attention 
to  the  lethality  of  the  modified  warhead  and  impact  on 
weapon  accuracy  of  end-to-end  flight  testing  in  more 
challenging  environments  and  conditions  when  moving  to 
the  Government  Confidence  Test  phase  of  the  program. 
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Ballistic  Missile  Defense  System  (BMDS) 
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USSTRATCOM  /  NMCC 


Aegis  BMD  -  Aegis  Ballistic  Missile  Defense  THAAD  -  Terminal  High-Altitude  Area  Defense  USNORTHCOM  -  U.S.  Northern  Command 
NMCC  -  National  Military  Command  Center  UEWR  -  Upgraded  Early  Warning  Radar  USPACOM  -  U.S.  Pacific  Command 

SM-3  -  Standard  Missile-3  USCENTCOM  -  U.S.  Central  Command  USSTRATCOM  -  U.S.  Strategic  Command 

USEUCOM  -  U.S.  European  Command 


Executive  Summary 

•  The  demonstrated  Ballistic  Missile  Defense  System  (BMDS) 
theater/regional  capability  increased  over  this  fiscal  year. 

-  The  BMDS  Operational  Test  Agency  and  the  Missile 
Defense  Agency  (MDA)  conducted  the  first  system-level 
operational  test  based  on  theater/regional  defense  designs 
applicable  to  multiple  Combatant  Commands  (CCMDs). 

-  The  new  capabilities  of  Aegis  Ballistic  Missile  Defense 
(BMD)  version  4.0  with  the  Standard  Missile-3  Block  IB 
interceptor  were  demonstrated. 

-  The  Fast  Exchange  ground  test  showed  additional  system 
interoperability  between  two  CCMDs. 

•  The  BMDS  homeland  defense  capability  also  increased 
over  this  fiscal  year.  The  MDA  conducted  Flight  Test 
Ground-based  lnterceptor-06b  (FTG-06b)  in  June  2014.  This 
first  successful  intercept  by  the  Capability  Enhancement  II 
exoatmospheric  kill  vehicle  allowed  production  manufacturing 
to  resume. 

•  In  September  2013,  the  BMDS  Operational  Test  Agency 
and  the  MDA  conducted  Flight  Test  Operational-0 1 
(FTO-01).  The  FTO-01  test  mission  demonstrated  a  layered 
upper-tier  regional/theater  BMDS  defense  against  a  raid 

of  two  simultaneously-launched  and  threat-representative 
medium- range  ballistic  missiles  threatening  a  shared  defended 
area. 

-  No  additional  system-level  flight  testing  occurred  during 
FY14.  The  BMDS  system-level  flight  test  activity  in  FY14 
focused  on  analyzing  the  test  results  from  FTO-0 1  and 
planning  for  the  FTO-02  test  events  in  FY15. 


•  The  MDA  executed  numerous  ground  tests  and  also  conducted 
several  wargames  and  exercises. 

•  The  MDA  enhanced  the  Integrated  Master  Test  Plan  (IMTP) 
by  including  direct  linkage  between  the  BMDS  test  program 
and  future  capability  enhancements. 

•  FY14  capability  to  produce  BMDS-level  simulation-based 
performance  assessments  was  significantly  limited.  The 
MDA  should  increase  the  development  priority  and  associated 
funding  for  the  BMDS  high-fidelity,  end-to-end,  digital 
modeling  and  statistically  significant  simulation  capability. 

System 

•  The  BMDS  is  a  distributed  system  currently  comprised  of  five 
elements  (four  shooter  elements  and  one  command  and  control 
element)  and  six  sensor  systems  (five  radar  systems  and  one 
space-based  system). 

Elements 

•  Aegis  BMD,  including  Aegis  Ashore  (shooter) 

•  Ground-based  Midcourse  Defense  (GMD)  (shooter) 

•  Patriot  (shooter) 

•  Terminal  High-Altitude  Area  Defense  (THAAD)  (shooter) 

•  Command  and  Control,  Battle  Management,  and 
Communications  (C2BMC)  (command  and  control) 

Sensors 

•  AN/SPY- 1  Radar  (Aegis  BMD) 

•  AN/TPY-2  Forward-Based  Mode  (FBM)  Radar 

•  COBRA  DANE  Radar  Upgrade 

•  Upgraded  Early  Warning  Radars  (UEWRs) 
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•  Sea-Based  X-band  (SBX)  Radar 

•  Space-Based  Infrared  System/Defense  Support  Program 
(SBIRS/DSP) 

Mission 

•  The  U.S.  Strategic  Command  (USSTRATCOM)  synchronizes 
operational-level  global  missile  defense  planning  and 
operations  support  for  the  DOD. 

•  U.S.  Northern  Command  (USNORTHCOM),  U.S.  Pacific 
Command  (USPACOM),  U.S.  European  Command 
(USEUCOM),  and  U.S.  Central  Command  (USCENTCOM) 
employ  the  assets  of  the  BMDS  to  defend  U.S.  territory, 
deployed  forces,  and  allies  against  ballistic  missile  threats  of 
all  ranges. 

•  USSTRATCOM,  USNORTHCOM,  USEUCOM, 
USCENTCOM,  and  USPACOM  use  the  C2BMC  element  of 
the  BMDS  to  maintain  situational  awareness.  USEUCOM, 
USCENTCOM,  and  USPACOM  also  use  the  C2BMC  to 
provide  sensor  management  of  theater  AN/TPY-2  (FBM) 
radars. 

•  Combatant  Commanders  employ  Patriot  to  provide  theater 
defense  for  deployed  forces  against  short-  and  medium-range 
threats. 


Major  Contractors 

•  The  Boeing  Company 

-  BMDS  and  GMD  Integration  -  Huntsville,  Alabama 

•  Lockheed  Martin  Corporation 

-  Aegis  BMD,  Aegis  Ashore,  and  AN/SPY- 1 
Radar  -  Moorestown,  New  Jersey 

-  C2BMC  -  Gaithersburg,  Maryland 

-  SB1RS  -  Sunnyvale,  California 

-  THAAD  Weapon  System  and  Patriot 
Interceptors  -  Dallas,  Texas 

-  THAAD  Interceptors  -  Troy,  Alabama 

•  Northrop  Grumman  Corporation 

-  DSP  -  Redondo  Beach,  California 

-  GMD  Fire  Control  and 
Communications  -  Huntsville,  Alabama 

•  Orbital  Sciences  Corporation 

-  GMD  Booster  Vehicles  -  Chandler,  Arizona 

•  Raytheon  Company 

-  GMD  Exoatmospheric  Kill  Vehicle  and  Standard  Missile-3 
Interceptors  -  Tucson,  Arizona 

-  Patriot,  AN/TPY-2  Radar,  Cobra  Dane  Radar,  SBX 
Radar,  and  UEWRs/EWRs  -  Tewksbury,  Massachusetts 


Activity 

•  The  BMDS  Operational  Test  Agency  and  the  MDA  conducted 
the  first  operational  system-level  flight  test  (FTO-Ol)  in 
September  2013. 

-  The  FTO-Ol  test  mission  demonstrated  a  layered 
upper- tier  regional/theater  BMDS  defense  against  a  raid 
of  two  simultaneously-launched  and  threat-representative 
medium-range  ballistic  missiles  threatening  a  shared 
defended  area. 

-  No  additional  system-level  flight  testing  occurred 
during  FY14,  which  was  in  accordance  with  the 
DOT&E-approved  IMTP. 

-  The  next  system-level  operational  flight  test,  FTO-02,  is 
scheduled  for  FY15. 

-  The  BMDS  system-level  flight  test  activity  in  FY14 
focused  on  analyzing  the  test  results  from  FTO-0 1  and 
planning  for  the  FTO-02  test  events. 

•  The  MDA  conducted  FTG-06b  in  June  2014.  FTG-06b  was 
a  system-level  flight  test  of  the  BMDS  Homeland  Defense 
capability. 

•  During  FY14,  the  MDA  executed  four  major  ground  tests, 
including  examining  cross-CCMD  sharing  of  sensor  data 
in  its  Fast  Exchange  test.  The  MDA  also  conducted  several 
wargames  and  exercises. 

Assessment 

•  The  FTO-Ol  test  mission  demonstrated  a  layered 
theater/ regional  upper-tier  ballistic  missile  defense  using  a 
BMDS  instantiation  consisting  of  the  Aegis  BMD,  THAAD, 


C2BMC,  AN/TPY-2  (FBM),  and  SBIRS/DSP  elements.  The 
classified  DOT&E  February  2014  BMDS  Annual  Report, 
Appendix  E,  assesses  the  results  from  this  mission  and 
includes  six  key  findings  covering  system-level  performance, 
effectiveness,  and  suitability. 

-  Although  a  layered  defense  was  demonstrated  in  FTO-Ol, 
true  system  integration  was  not  demonstrated  due  to 
system  network  configuration  errors,  interoperability 
limitations,  and  component  failures. 

•  The  BMDS  theater/regional  capability  increased  over  this 
fiscal  year. 

-  FTO-0 1  demonstrated  an  operationally  representative 
scenario  based  on  five  theater/regional  defense 
designs  applicable  to  USPACOM,  USEUCOM,  and 
USCENTCOM. 

-  Aegis  BMD  version  4.0,  together  with  the  SM-3  Block  IB 
interceptor,  brings  new  capabilities  to  theater/regional 
engagements  that  successfully  completed  operational 
testing.  Details  of  this  testing  and  the  Aegis  BMD/SM-3 
Block  IB  assessment  are  reported  in  the  individual  article 
on  Aegis  BMD  later  in  this  section. 

-  The  Fast  Exchange  ground  test  showed  additional  system 
interoperability  between  USEUCOM  and  USCENTCOM 
areas  of  responsibility.  System-level  ground  testing  is 
reported  in  the  individual  article  on  C2BMC  later  in  this 
section. 

•  The  BMDS  homeland  defense  capability  increased  over 
this  fiscal  year.  The  first  successful  intercept  by  the 
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Capability  Enhancement  II  exoatmospheric  kill  vehicle 
allowed  production  manufacturing  to  resume.  The  FTG-06b 
assessment  is  reported  in  the  individual  article  on  GMD  later 
in  this  section. 

•  The  MDA,  in  collaboration  with  DOT&E,  updated  the 
FY13  version  of  the  IMTP  to  incorporate  BMDS  element 
maturity,  program  modifications,  and  fiscal  constraints.  They 
also  enhanced  the  IMTP  by  including  direct  linkage  between 
the  BMDS  test  program  and  future  capability  enhancements  as 
defined  by  the  BMDS  Phased  Implementation  Plan. 

•  FY14  capability  to  produce  BMDS-level  performance 
assessments  was  significantly  limited.  The  MDA  is  developing 
a  new  high-fidelity,  end-to-end,  digital  performance 
assessment  modeling  and  simulation  capability  for  the  BMDS. 
This  effort  began  in  FY12  and  is  currently  in  the  requirements 
definition  phase.  Requirements  definition  is  funded  through 
FY15.  In  addition,  three  Small  Business  Innovative  Research 
contracts  were  awarded  in  FY14  to  explore  a  simulation 


capability  to  produce  statistically  significant  run  sets.  The 
MDA  will  assess  future  funding  needs  for  the  implementation 
of  the  final  design  requirements  as  these  initiatives  proceed 
through  the  remainder  of  the  development  program. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  MDA  has 
addressed  previous  system-level  recommendations. 

•  FY 14  Recommendations.  The  MDA  should: 

1.  Address  recommendations  made  in  the  DOT&E  FTO-Ol 
assessment  found  in  the  classified  DOT&E  February  2014 
BMDS  Annual  Report,  Appendix  E. 

2.  Increase  the  development  priority  and  associated  funding 
for  the  BMDS  simulation-based  performance  assessment 
capability,  as  the  ability  to  produce  high-fidelity  and 
statistically  significant  BMDS-level  performance 
assessments  is  critical. 
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Aegis  Ballistic  Missile  Defense  (Aegis  BMD) 


Executive  Summary 

•  The  Aegis  Ballistic  Missile  Defense  (BMD)  program  conducted 
one  intercept  mission  in  FY14  and  one  in  early  FY15.  Two 
ballistic  missile  targets  were  intercepted,  and  two  anti-air 
warfare  targets  were  engaged. 

•  In  FY14,  Aegis  BMD  completed  the  IOT&E  flight  test 
program  for  the  Aegis  BMD  4.0  system  and  Standard  Missile-3 
(SM-3)  Block  IB  guided  missile,  in  accordance  with  the 
DOT&E-approved  Integrated  Master  Test  Plan. 

•  The  final  IOT&E  flight  test  was  a  successful  intercept  of  a 
medium-range  ballistic  missile,  and  was  the  fifth  successful 
intercept  of  the  Aegis  BMD  4.0/SM-3  Block  IB  system. 

•  In  the  first  live  fire  event  using  Aegis  Baseline  9. BO  (i.e.,  Aegis 
BMD  5.0),  Aegis  Ashore  (a  future  land-based  component  of  the 
European  Phased- Adaptive  Approach  (EPAA))  demonstrated 

a  capability  to  fire,  control,  establish  uplink/  downlink 
communication,  provide  guidance  commands,  and  provide 
target  information  to  an  SM-3  Block  IB  guided  missile. 

•  The  Aegis  BMD  program  continued  testing  of  the  Aegis 
BMD  5.0  system  (Capability  Upgrade  version.  Baseline 
9. Cl)  by  conducting  two  tests  in  early  FY15:  a  simulated 
engagement  of  a  medium-range  ballistic  missile  and  a  live 
intercept  of  a  short-range  ballistic  missile  while  simultaneously 
engaging  a  raid  of  two  subsonic  cruise  missiles. 

•  Aegis  BMD  continued  to  improve  interoperability  with  other 
Ballistic  Missile  Defense  System  (BMDS)  elements  and 
sensors  during  flight  and  ground  testing  in  FY14. 

•  Hardware-in-the-loop  (HWIL)  ground  testing  demonstrated 
Aegis  BMD  capability  to  contribute  to  theater,  regional 
(including  EPAA),  and  strategic-level  defense  missions 
spanning  a  range  of  ballistic  missile  defense  scenarios. 

System 

•  Aegis  BMD  is  a  sea-based  missile  defense  system  that  employs 
the  multi-mission  shipboard  Aegis  Weapon  System,  with 
improved  radar  and  new  missile  capabilities  to  engage  ballistic 
missile  threats.  Capabilities  of  Aegis  BMD  include: 

-  Computer  program  modifications  to  the  AN/SPY- 1  radar 
for  long-range  surveillance  and  track  (LRS&T)  of  ballistic 
missiles  of  all  ranges. 

-  A  modified  Aegis  Vertical  Launching  System,  which  stores 
and  fires  SM-3  Block  IA  and  Block  IB  guided  missiles  (on 
select  ships),  and  modified  SM-2  Block  IV  guided  missiles 
(on  select  ships). 

-  SM-3  Block  1A  and  Block  IB  guided  missiles,  which  use 
a  maneuverable  kinetic  warhead  to  accomplish  midcourse 
engagements  of  short-,  medium-,  and  intermediate-range 
ballistic  missiles. 

-  Modified  SM-2  Block  IV  guided  missiles,  which  provide 
terminal  engagement  capability  against  short-range  ballistic 
missiles. 


•  Aegis  Ashore  is  a  land-based  version  of  Aegis  BMD,  with  an 
AN/SPY- 1  radar  and  Vertical  Launching  System  to  enable 
engagements  against  medium-  and  intermediate-range  ballistic 
missiles  with  SM-3  guided  missiles.  Once  it  is  deployed 

to  Romania  in  2015,  Aegis  Ashore  will  become  the  central, 
land-based  component  of  the  second  phase  of  the  EPAA  for 
the  defense  of  Europe. 

•  Aegis  BMD  and  Aegis  Ashore  are  capable  of  performing 
autonomous  missile  defense  operations  and  operations  that 
exploit  networked  sensor  information;  they  can  send/receive 
cues  to/from  other  BMDS  sensors  through  tactical  datalinks. 

Mission 

The  Navy  can  accomplish  three  missile  defense-related  missions 

using  Aegis  BMD: 

•  Defend  deployed  forces  and  allies  from  short-  to 
intermediate-range  theater  ballistic  missile  threats 

•  Provide  forward-deployed  radar  capabilities  to  enhance 
defense  against  ballistic  missile  threats  of  all  ranges  by 
sending  cues  or  target  track  data  to  other  elements  of  the 
BMDS 

•  Provide  all  short-  to  long-range  ballistic  missile  threat  data 
to  the  Command  and  Control,  Battle  Management,  and 
Communications  (C2BMC)  system  for  dissemination  to 
Combatant  Commanders’  headquarters  to  ensure  situational 
awareness 

Major  Contractors 

•  Aegis  BMD  Weapon  System:  Lockheed  Martin  Corporation, 
Mission  Systems  and  Training  -  Moorestown,  New  Jersey 

•  AN/SPY- 1  Radar:  Lockheed  Martin  Corporation,  Mission 
Systems  and  Training  -  Moorestown,  New  Jersey 

•  SM-3  Missile:  Raytheon  Company,  Missile 
Systems  -  Tucson,  Arizona 
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Activity 

•  The  MDA  completed  the  IOT&E  flight-testing  phase  for 
Aegis  BMD  4.0  and  SM-3  Block  IB  guided  missiles  with 
the  conclusion  of  Flight  Test  Standard  Missile-21  (FTM-21) 
(FY13),  FTM-22,  and  Flight  Test  Other- 18  (FTX-18).  All 
flight  testing  was  accomplished  in  accordance  with  the 
DOT&E-approved  Integrated  Master  Test  Plan. 

•  The  Aegis  BMD  program  executed  modeling  and  simulation 
runs-for-the-record  in  FY14  to  supplement  flight  test  data  and 
aid  in  the  assessment  of  Aegis  BMD  4.0  system  effectiveness. 
DOT&E  published  a  classified  IOT&E  report  on  Aegis 
BMD  4.0  and  SM-3  Block  IB  operational  effectiveness  and 
operational  suitability  in  December  2014  to  support  the 
Full-Rate  Production  decision  for  SM-3  Block  IB  missiles. 

The  Full-Rate  Production  decision  is  planned  for  FY15. 

•  Although  the  program  completed  FOT&E  for  the  Aegis 
BMD  3.6.1  system  in  FY11,  the  program  continued  to  use 
variants  of  the  Aegis  BMD  3.6  system  (i.e.,  3.6. 1  and  3.6.3)  in 
BMDS-level  tests  in  FY14  to  assess  system-level  engagement 
capability  and  interoperability. 

•  The  Aegis  BMD  program  conducted  one  intercept  mission 
in  FY14  and  one  in  early  FY15.  Two  ballistic  missile  targets 
were  intercepted,  and  two  anti-air  warfare  targets  were 
engaged. 

-  During  FTM-22  in  October  2013,  an  Aegis  BMD  4.0.2 
cruiser  intercepted  a  medium-range,  separating  target 
with  an  SM-3  Block  IB  guided  missile.  The  FTM-22 
engagement  was  the  fifth  successful  intercept  mission 
conducted  with  the  Aegis  BMD  4.0  system  with  an 
SM-3  Block  IB  guided  missile,  and  the  second  of  two 
IOT&E  flight  test  missions  (the  other  being  FTM-21  in 
September  2013).  The  FTM-22  engagement  was  also  the 
first  intercept  of  a  medium-range  target  with  the  Aegis 
BMD  4.0  system  and  an  SM-3  Block  IB  guided  missile. 

-  FTM-25  in  November  2014  was  an  integrated  air  and 
missile  defense  mission  wherein  an  Aegis  Baseline  9. Cl 
(i.e.,  Aegis  BMD  5.0  with  Capability  Upgrade)  destroyer  in 
integrated  air  and  missile  defense  priority  mode  detected, 
tracked,  and  performed  an  SM-3  Block  IB  intercept  of 

a  short-range  ballistic  missile,  while  simultaneously 
conducting  an  anti-air  warfare  raid  engagement  against  two 
subsonic  cruise  missile  surrogates. 

•  Aegis  BMD  ships  and  Aegis  Ashore  participated  in  simulated 
engagements  using  live  targets,  a  controlled  vehicle  test  with 
a  live  interceptor,  and  a  multi-element  test  event  with  the 
AN/SPY-1  radar  acting  as  a  forward-based  sensor  in  FY 14. 

-  During  FTX-18  in  January  2014,  an  Aegis  BMD  4.0.2 
destroyer  detected,  tracked,  and  conducted  simulated 
SM-3  Block  IB  engagements  against  three,  short-range 
ballistic  missile  targets  in  a  raid  scenario.  FTX-18  was 
the  last  of  three  Aegis  BMD  4.0  IOT&E  test  missions.  An 
Aegis  BMD  3.6.1  ship  and  Aegis  Ashore  (Aegis  Baseline  9 
Lab  at  Surface  Combat  Systems  Center,  Wallops  Island, 
Virginia)  also  participated  in  FTX-18  for  data  collection 
purposes  and  to  exercise  simulated  engagements  using 
remote  data  sent  from  the  Aegis  BMD  4.0.2  ship. 
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-  In  May  2014,  Aegis  Ashore  Control  Test  Vehicle-01 
(AACTV-01)  demonstrated  the  capability  of  the  Aegis 
Ashore  test  site  at  the  Pacific  Missile  Range  Facility  to  fire, 
control,  establish  uplink/downlink  communication,  provide 
guidance  commands,  and  provide  target  information  to 

an  SM-3  Block  IB  guided  missile.  AACTV-01  was  the 
first  live  fire  event  using  Aegis  Baseline  9. B0  (i.e.,  Aegis 
BMD  5.0)  and  the  first  SM-3  guided  missile  firing  from 
Aegis  Ashore. 

-  In  June  2014,  an  Aegis  BMD  3.6. 1  destroyer  participated 
in  Flight  Test  Ground-Based  Interceptor-06b  (FTG-06b) 
to  provide  LRS&T  support  to  the  Ground-Based 
Midcourse  Defense  (GMD)  system.  The  ship  acquired  the 
intermediate-range  ballistic  missile  target  and  transmitted 
track  information  over  Link  16  to  support  the  generation 
of  GMD’s  weapon  task  plan  and  a  cue  to  the  Sea-Based 
X-band  radar. 

-  In  Flight  Test  Other-20  (FTX-20)  in  October  2014,  a 
destroyer  with  Aegis  Baseline  9. Cl  software  conducted 
a  simulated  SM-3  Block  IB  engagement  of  a  separating 
medium-range  ballistic  missile. 

•  In  FY14,  Aegis  BMD  ships  and  HWIL  facilities  also 

participated  in  several  flight  and  ground  tests  to  assess 

Aegis  BMD  4.0.2  and  3. 6. 1/3. 6. 3  system  functionality  and 

interoperability  with  the  BMDS. 

-  Ground  Test  lntegrated-04e  (GTl-04e)  Part  la 
runs-for-the-record  in  October  2013  explored 

theater/ regional  defense  capabilities  (beyond  those  tested 
in  GTI-04e  Part  1  in  FY13)  using  updated  software  builds 
for  AN/TPY-2  (Forward  Based  Mode  (FBM)),  C2BMC, 
and  Terminal  High- Altitude  Area  Defense  (THAAD),  in 
addition  to  Aegis  BMD  4.0.2  and  3.6.3. 

-  Aegis  BMD  participated  in  Fast  Phoenix  in 
December  2013,  which  the  Missile  Defense  Agency 
(MDA)  executed  in  the  U.S.  Pacific  Command  area 
of  responsibility,  while  simulating  U.S.  European 
Command  and  U.S.  Central  Command  interoperability 
architectures.  The  test’s  purpose  was  to  assess  a  new 
communications  architecture  between  Aegis  BMD 
ships  using  the  Link  Monitoring  Management  Tool  and 
C2BMC  S6.4  Maintenance  Release  2.  Fast  Phoenix 
used  both  operational  and  HWIL  assets  and  operational 
communications. 

-  GTI-04e  Part  2  in  April  and  May  2014  tested  the 
engagement  capabilities  of  fielded  and  to-be-fielded  missile 
defense  elements  and  sensors  against  ballistic  missiles  of 
all  ranges  in  a  HWIL  environment.  Participants  included 
Aegis  BMD  4.0.2  and  3.6.3  (laboratory  sites),  C2BMC, 
Patriot,  THAAD,  Space-Based  Infrared  System,  AN/TPY-2 
(FBM),  and  GMD. 

-  Fast  Fire,  which  the  MDA  conducted  as  part  of  GTl-04e 
Part  2  in  May  2014,  tested  the  capability  of  Aegis 
BMD  4.0.2  to  support  its  designed-to  maximum  number 
of  simultaneous  ballistic  missile  and  anti-air  warfare 
engagements  and  control  all  standard  missiles  in  those 
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engagements.  An  AN/TPY-2  (FBM)  radar  provided 
forward-based  radar  data  to  support  the  engagements. 

-  Aegis  BMD  participated  in  Fast  Exchange  in  both  FiWIL 
(June  2014)  and  distributed  (August  2014)  representations 
to  assess  cross- Area  of  Responsibility  engagements. 
Additional  participants  included  AN/TPY-2  (FBM), 
TF1AAD,  Patriot,  C2BMC,  and  Space-Based  Infrared 
System. 

•  Early  developmental  testing  of  the  SM-3  Block  IIA  guided 
missile  began  in  FY14. 

-  In  October  2013,  the  SM-3  Cooperative  Development 
Propulsion  Test  Vehicle-01  (SCDPTV-01)  flight  test  fired 
an  SM-3  Block  IIA  guided  missile.  The  missile  included  a 
live  Mk  72  Mod  2  booster  with  an  inert  21 -inch-diameter 
mass  equivalent  upper-stage  assembly  in  the  Mk  29  Mod  0 
lightweight  canister,  specifically  developed  for  the  SM-3 
Block  IIA  guided  missile.  All  assets  under  test  performed 
as  designed.  SCDPTV-01  was  a  follow-on  test  from  a 
restrained  firing  conducted  in  FY13,  and  the  second  of 

a  series  of  six  test  events  to  validate  missile  and  canister 
designs  of  the  next  variant  of  the  SM-3. 

Assessment 

•  Flight  testing  and  supporting  modeling  and  simulation 
demonstrated  that  Aegis  BMD  4.0  has  the  capability  to 
engage  and  intercept  non-separating,  simple-separating,  and 
complex-separating  ballistic  missiles  in  the  midcourse  phase 
with  SM-3  Block  IB  guided  missiles.  Flowever,  flight  testing 
and  modeling  and  simulation  did  not  test  the  full  range  of 
expected  threat  types,  threat  ground  ranges,  and  threat  raid 
sizes.  Details  can  be  found  in  the  classified  December  2014 
Aegis  BMD  IOT&E  Report. 

•  Reliability  and  maintainability  data  from  FY14,  in 
combination  with  data  collected  during  a  maintenance 
demonstration  and  previous  flight  testing,  suggest  that  overall 
Aegis  BMD  4.0  Weapon  System  availability  is  adequate 

for  the  midcourse  defense  mission  against  short-  and 
medium-range  ballistic  missiles.  Testing  showed  that 
improvements  in  Aegis  BMD  hardware  reliability  are  needed, 
although  the  impact  on  operational  availability  was  not 
significant  due  to  the  low  repair  times. 

•  The  limited  number  of  SM-3  Block  IB  firings  (nine)  and 
the  two,  third-stage  rocket  motor  (TSRM)  failures  (FTM-16 
Event  2  in  FY11  and  the  second  missile  failure  in  FTM-21 
in  FY13)  lower  certainty  in  overall  SM-3  Block  IB  missile 
reliability.  The  program  addressed  and  tested  a  correction  for 
the  first  of  the  SM-3  TSRM  problems  when  it  modified  the 
TSRM’s  inter-pulse  delay  time.  The  Aegis  BMD  program 
has  exercised  the  new  inter-pulse  delay  without  incident  in 
three  flight  tests  and  a  number  of  ground-based  static  firings. 
The  correction,  however,  did  not  prevent  the  TSRM  failure 

in  the  second  of  two  salvo-launched  SM-3  Block  IB  guided 
missiles  in  FTM-21,  which  also  suffered  a  reliability  failure  of 
the  TSRM  aft  nozzle  area  during  second  pulse  operations  of 
the  two-pulse  motor  (the  first  missile  had  already  achieved  a 


successful  intercept).  The  MDA  established  a  Failure  Review 
Board  (FRB)  to  determine  the  root  cause  of  this  failure  and 
the  Board  has  uncovered  enough  evidence  to  determine  that  a 
re-design  is  needed  for  the  TSRM  nozzle.  The  program  has  a 
preliminary  design  for  the  new  nozzle,  and  began  the  ground 
testing  of  new  design  concepts  in  FY14.  The  new  design 
will  be  retrofittable  into  current  SM-3  Block  IA  and  Block  IB 
missiles. 

•  Flight  testing  and  modeling  and  simulation  have  demonstrated 
the  Aegis  BMD  4.0  capability  to  perform  the  LRS&T  mission, 
albeit  with  only  a  single  threat.  Additionally,  the  FTG-07 
mission  in  FY13  highlighted  the  need  to  further  explore  and 
refine  tactics,  techniques,  and  procedures  (TTPs)  for  the 
transmission  and  receipt  of  Aegis  BMD  track  data  for  GMD 
use. 

•  Testing  of  the  Aegis  BMD  4.0  system  did  not  evaluate 
automated  engagement  coordination  in  flight  testing;  due  to 
lack  of  ship  availability.  However,  the  MDA  tested  it  during 
Ground  Test  Focused-04e  and  it  will  be  operationally  tested 
during  FOT&E. 

•  The  program  demonstrated  that  Aegis  Ashore  can  fire, 
detect,  and  control  an  SM-3  Block  IB  guided  missile  during 
AACTV-0 1 .  This  is  an  important  first  step  toward  proving  that 
Aegis  Ashore  can  perfonn  missile  defense  operations  similar 
to  those  on  an  Aegis  BMD  ship  for  the  defense  of  Europe  as 
part  of  EPAA  Phase  2.  An  engagement  of  a  ballistic  missile 
target  by  Aegis  Ashore  will  take  place  in  Event  1  of  Flight  Test 
Operational-02  (FTO-02)  in  FY15. 

•  During  FTX-20,  an  Aegis  Baseline  9. Cl  destroyer 
(Aegis  BMD  5.0  with  Capability  Upgrade)  successfully 
detected,  tracked,  and  conducted  a  simulated  engagement  of  a 
separating  medium-range  ballistic  missile  target. 

•  During  FTM-25,  an  Aegis  Baseline  9.Cl-configured  destroyer, 
operating  in  integrated  air  and  missile  defense  priority  mode, 
intercepted  a  short-range  ballistic  missile  target  using  an  SM-3 
Block  IB  guided  missile  while  simultaneously  engaging  two 
subsonic  cruise  missile  targets  using  two  SM-2  Block  IIIA 
missiles. 

•  The  MDA  continues  to  utilize  Aegis  BMD  ships  and  HW1L 
representations  of  the  Aegis  BMD  4.0  and  3.6  variants, 
which  has  helped  to  refine  TTPs  and  overall  interoperability 
of  the  system  with  the  BMDS.  However,  the  test  events 
routinely  demonstrated  that  inter  element  coordination  and 
interoperability  are  still  in  need  of  improvement. 

Recommendations 

•  Status  of  Previous  Recommendations. 

-  The  program  has  not  addressed  the  first  two 

recommendations  (out  of  five)  from  FY13  to  conduct: 

■  Flight  testing  of  the  Aegis  BMD  4.0  remote  authorized 
engagement  capability  against  a  medium-range  ballistic 
missile  or  intermediate-range  ballistic  missile  target  using 
an  SM-3  Block  IB  guided  missile.  FTO-02  Event  2, 
scheduled  for  4QFY15,  is  planned  to  demonstrate  this 
capability 
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■  Operationally  realistic  testing  that  exercises  Aegis 
BMD  4.0’s  improved  engagement  coordination  with 
THAAD  and  Patriot 

-  The  program  addressed  the  third  recommendation  from 
FY13  to  continue  to  assess  an  Aegis  BMD  4.0  intercept 
mission  during  which  the  ship  simultaneously  engages 
an  anti-air  warfare  target  to  verify  BMD/anti-air  warfare 
capability,  when  it  conducted  the  FTM-25  mission  using 
Aegis  Baseline  9.C1. 

-  The  program  partially  addressed  the  fourth 
recommendation  from  FY13  to  use  the  FRB  process  to 
identify  the  failure  mechanism  responsible  for  the  FTM-21 
second  missile  failure  and  determine  the  underlying 

root  cause  that  may  be  common  to  both  the  FTM- 1 6 
Event  2  and  FTM-21  second  missile  failures.  The  MDA 
established  an  FRB  following  FTM-21  and,  although  it 
is  still  ongoing,  preliminary  findings  from  the  FRB  have 
pointed  to  a  similar  root  cause,  prompting  the  program  to 
begin  a  re-design  of  the  TSRM  nozzle. 

-  The  program  partially  addressed  the  fifth  recommendation 
from  FY13  to  deliver  sufficient  Aegis  BMD  4.0  validation 
data  and  evidence  to  support  BMDS  modeling  and 
simulation  verification,  validation,  and  accreditation 
(VV&A)  of  the  Aegis  F1W1L  and  digital  models.  They  did 
so  when  the  Commander,  Operational  Test  and  Evaluation 
Force  provided  VV&A  evidence  for  the  digital  models  used 
for  element-level  performance  analyses  in  support  of  the 


operational  assessment  of  the  Aegis  BMD  4.0  system  with 
SM-3  Block  IB  guided  missiles.  Aegis  BMD  provided 
V&V  data  from  element  post-flight  reconstruction  events 
for  FTM- 16  Event  2,  FTM- 18,  and  FTM-21,  based  on 
BMDS  Operational  Test  Agency  performance  parameters 
and  acceptability  criteria  as  evidence  supporting 
accreditation  of  Aegis  BMD  HWIL  models  participating 
in  BMDS  level  ground  testing.  The  BMDS  Operational 
Test  Agency  is  reviewing  the  data  for  an  accreditation 
recommendation. 

FY14  Recommendations.  The  program  should: 

1 .  Conduct  flight  tests  or  high-fidelity  modeling  and 
simulation  analyses  to  demonstrate  the  Aegis  BMD  4.0 
system’s  capability  to  perform  LRS&T  of  a  raid  of 
long-range  threats. 

2.  Determine  the  appropriate  LRS&T  TTPs  for  the 
transmission  and  receipt  of  Aegis  BMD  4.0  track  data  for 
GMD  use. 

3.  Ensure  that  sufficient  flight  testing  of  the  Aegis  Baseline 
9. Cl  system  is  conducted  to  allow  for  VV&A  of  the 
modeling  and  simulation  suite  to  cover  the  full  design  to 
Aegis  BMD  battlespace  of  threat  ballistic  missiles. 

4.  Conduct  sufficient  ground  and  flight  testing  of  the 
re-designed  SM-3  Block  IB  TSRM  nozzle  after  completion 
and  installation  of  the  new  design  concept  to  prove  the  new 
design  works  under  the  most  stressing  operational  flight 
conditions. 
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Command  and  Control,  Battle  Management,  and 
Communications  (C2BMC)  System 


Executive  Summary 

•  The  Missile  Defense  Agency  (MDA)  continued  to 
demonstrate  an  increased  maturity  of  Command  and  Control, 
Battle  Management,  and  Communications  (C2BMC) 

Spiral  6.4  (S6.4)  software  during  FY14. 

-  Ground  and  flight  testing  demonstrated  automated 
management  of  multiple  AN/TPY-2  Forward-Based  Mode 
(FBM)  sensors.  Ground  testing  further  demonstrated 
tasking  of  the  AN/TPY-2  radar  in  the  presence  of 
post-intercept  debris. 

-  C2BMC  demonstrated  the  capability  for  Combatant 
Command  (CCMD)  sensor  managers  to  direct  AN/TPY-2 
(FBM)  radars  to  execute  focused  search  plans  or  respond 
to  precision  cues. 

-  C2BMC  demonstrated  timely  and  accurate  forwarding  of 
radar  track  data  during  numerous  ground  and  flight  tests. 

•  Ground  and  flight  testing  have  identified  numerous 
C2BMC  S6.4  deficiencies  in  situational  awareness  and 
system-level  interoperability.  Furthermore,  C2BMC  S6.4  does 
not  have  an  engagement  management  capability.  The  MDA 

is  relying  on  C2BMC  Spiral  8.2  (S8.2)  (FY17-18)  to  address 
these  deficiencies. 

•  During  FY 14,  the  MDA  and  Red  Teams  from  the  Threat 
Systems  Management  Office  conducted  three  cyber 
assessments  of  future  C2BMC  software  spirals  using  the 
newly  created  DOD  Enterprise  Cyber  Range  Environment 
(DECRE).  In  response,  the  MDA  is  examining  solutions  and 
procedures  that  can  reduce  the  likelihood  of  occurrence  or 
effect  of  cyber  penetration. 

System 

•  The  C2BMC  system  is  a  CCMD  interface  to  the  Ballistic 
Missile  Defense  System  (BMDS).  More  than  70  C2BMC 
workstations  are  fielded  at  U.S.  Strategic,  U.S.  Northern, 

U.S.  European,  U.S.  Pacific,  and  U.S.  Central  Commands 
(USSTRATCOM,  USNORTHCOM,  USEUCOM, 

USPACOM,  and  USCENTCOM);  numerous  Army  Air  and 
Missile  Defense  Commands;  Air  and  Space  Operations 
Centers;  and  other  supporting  warfighter  organizations. 

-  The  current  C2BMC  provides  CCMDs  and  other  senior 
national  leaders  with  situational  awareness  on  BMDS 
status,  system  coverage,  and  ballistic  missile  tracks  by 
displaying  selective  data  from  the  Global  Communications 
Network  for  strategic/national  missile  defense  and  from 
the  Joint  Tactical  Information  Distribution  System  Link  16 
for  theater/regional  missile  defense. 

-  The  C2BMC  also  provides  upper  echelon  deliberate 
planning  at  the  CCMD  and  component  level.  BMDS 


elements  (Aegis  Ballistic  Missile  Defense  [BMD], 
Ground-based  Midcourse  Defense  [GMD],  Patriot,  and 
Terminal  High-Altitude  Area  Defense  [THAAD])  use  their 
own  command  and  control  battle  management  systems  and 
mission  planning  tools  for  stand-alone  engagements. 

-  The  C2BMC  S6.4  suite  provides  command  and  control  for 
the  AN/TPY-2  FBM  radar.  The  S6.4  Global  Engagement 
Manager  Suite  provides  updated  sensor  management,  track 
processing,  and  reporting. 

•  Through  the  Global  Communications  Network,  the  C2BMC 
forwards  AN/TPY-2  (FBM)  and  AN/SPY- 1  tracks  to 
GMD.  Additionally,  through  the  Joint  Tactical  Information 
Distribution  System  Link  16,  it  forwards  AN/TPY-2  (FBM) 
tracks  for  THAAD  and  Patriot  cueing  and  Aegis  BMD 
engagement  support. 

•  The  C2BMC  S8.2  is  intended  to  improve  and  expand  on 
S6.4  capabilities  as  the  next  step  toward  integrated  sensor 
management  and  engagement  coordination. 

Mission 

USSTRATCOM,  USNORTHCOM,  USEUCOM,  USPACOM, 

and  USCENTCOM  use  C2BMC  to  support  ballistic  missile 
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defense  engagements.  More  specifically,  Combatant 
Commanders  use  C2BMC  for: 

•  Deliberate  and  dynamic  planning 

•  Situational  awareness 

•  Track  management 

•  AN/TPY-2  (FBM)  sensor  management  and  control 

•  Engagement  monitoring 


•  Data  exchange  between  C2BMC  and  BMDS  elements 

•  Network  management 

Major  Contractor 

Lockheed  Martin  Corporation,  Information  Systems  and  Global 
Solutions  -  Gaithersburg,  Maryland 


Activity 

•  The  MDA  conducted  C2BMC  testing  during  FY14  in 
accordance  with  the  DOT&E-approved  Integrated  Master  Test 
Plan. 

•  C2BMC  S6.4  Maintenance  Release  2  (MR2)  participated  in  the 
Flight  Test  Operational-Ol  (FTO-Ol)  BMDS-level  flight  test  in 
September  2013. 

•  The  MDA  executed  Fast  Phoenix  in  December  2013, 
a  distributed  ground  test  for  USPACOM  testing  a  new 
communications  architecture  between  Aegis  BMD 
ships  using  the  Link  Monitoring  Management  Tool  and 
C2BMC  S6.4  MR2.  Fast  Phoenix  used  both  operational 
and  hardware-in-the-loop  (HW1L)  assets  and  operational 
communications. 

•  C2BMC  participated  in  Ground  Test  lntegrated-04e  (GTI-04e) 
Part  la  and  GTI-04e  Part  2,  during  which  C2BMC  S6.4  MR2 
managed  multiple  AN/TPY-2  (FBM)  radars. 

-  Part  la  provided  data  for  theater/regional  interoperability 
assessments  and  focused  on  sensor  debris  mitigation  (DM) 
functionality  in  defense  of  USEUCOM  and  USCENTCOM. 
The  MDA  conducted  Phase  I  in  April  2013  with  DM 
disabled  and  Phase  II  in  October  2013  with  DM  enabled. 

-  Part  2  took  place  in  March  2014  and  focused  on  supporting 
the  defense  of  USPACOM  and  USNORTHCOM.  The  test 
exercised  several  updates  to  C2BMC  S6.4  MR2,  including 
the  tasking/managing  of  the  Kyoga-Misaki  AN/TPY-2 
(FBM)  radar,  boost  phase  cueing  by  the  AN/TPY-2  (FBM) 
radars,  and  updates  to  the  focused  search  plans  and  threat 
enumerations.  Part  2  also  demonstrated  interoperability 
with  Aegis  BMD  in  simulations  for  USCENTCOM. 

•  In  lune  2014,  C2BMC  S6.4  MR2  participated  in  Flight 
Test  Ground-Based  lnterceptor-06b  (FTG-06b).  C2BMC 
provided  situational  awareness  and  forwarded  track  data  from 
Aegis  BMD  and  Link  16  data  to  the  GMD  Fire  Control. 

•  C2BMC  S6.4  MR2  participated  in  the  Fast  Exchange  ground 
tests  in  both  HWIL  (June  2014)  and  fielded  (August  2014) 
representations.  C2BMC  managed  three  AN/TPY-2  (FBM) 
radars  in  a  cross-Area  of  Responsibility  environment. 

•  During  FY14,  the  MDA  and  Red  Teams  from  the  Threat 
Systems  Management  Office  conducted  three  cyber 
assessments  of  future  C2BMC  software  spirals  using  the 
newly  created  DECRE.  The  first  assessment  examined  cyber 
effects  that  previous  Red  Teams  were  in  a  position  to  deliver 
on  operational  networks  during  an  USEUCOM  assessment. 
Each  subsequent  event  added  additional  network  elements  and 
operational  realism. 


Assessment 

•  The  classified  DOT&E  February  2014  BMDS  Annual  Report, 
Appendix  E,  assesses  the  results  from  the  FTO-01  mission  and 
includes  six  key  findings  covering  C2BMC  participation  and 
system-level  performance,  effectiveness,  and  suitability. 

•  Fast  Phoenix  demonstrated  accurate  and  timely  data  sharing 
over  Link  16  between  Aegis  BMD  and  C2BMC  using  the 
Link  Monitoring  Management  Tool  and  C2BMC  Air  Defense 
Systems  Integrator. 

•  During  the  GTI-04e  Parts  la  and  2,  C2BMC  S6.4  MR2 
received  AN/TPY-2  (FBM)  and  Link  16  data  and  forwarded 
system  tracks  to  Link  16.  C2BMC  provided  situational 
awareness  and  demonstrated  interoperability  with  theater 
BMDS  elements.  It  controlled  two  AN/TPY-2  (FBM)  radars 
for  tasking  by  USEUCOM,  USPACOM,  USNORTHCOM,  and 
USCENTCOM.  Further,  C2BMC  demonstrated  boost  phase 
cue  capabilities  when  managing  two  AN/TPY-2  radars  for 
USPACOM.  Testing  uncovered  problems  with  missile-threat 
acquisition  using  focused  search  plans  designed  for  the  new 
USPACOM  radar  and  new  threat  characteristics.  The  MDA 

is  investigating  improvements  to  AN/TPY-2  radar  search-plan 
designs  and  C2BMC  sensor-tasking  algorithms  to  address 
these  problems. 

•  The  C2BMC  S6.4  MR2  performed  nominally  during  the 
FTG-06b  mission.  It  provided  situational  awareness  and 
forwarded  radar  track  information  as  designed.  However, 
it  did  not  receive  the  expected  Sea-Based  X-band 
radar-generated  engagement  hit  assessment  from  the  GMD  fire 
control  component.  The  MDA  is  investigating  why  this  event 
did  not  occur. 

•  Fast  Exchange  demonstrated  passing  of  pertinent, 
geographically-filtered  sensor  track  data  between  BMDS 
elements  in  USEUCOM  and  USCENTCOM.  Two  C2BMC 
suites  sent  and  received  AN/TPY-2  (FBM)  data  over  Link  16 
involving  the  Aegis  BMD,  THAAD,  and  Space-Based  Infrared 
System  elements. 

•  The  Red  Team  demonstrated  a  number  of  potential  cyber 
vulnerabilities  during  the  DECRE  events.  More  importantly, 
MDA  engineers  were  given  access  to  all  Red  Team  activities 
during  the  event  execution.  They  benefited  greatly  from 
first-hand  observations  and  the  opportunity  to  query  the 
Red  Team  regarding  steps  to  preclude  invasive  activities.  In 
response,  the  MDA  is  examining  solutions  and  procedures 
that  can  reduce  the  likelihood  of  occurrence  or  effect  of  cyber 
penetration.  Further,  the  MDA  is  actively  planning  additional 
cyber  assessments  using  the  DECRE  environment. 
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Recommendations 

•  Status  of  Previous  Recommendations.  The  MDA  addressed 
9  of  the  previous  1 1  recommendations.  The  MDA  continues 
to  make  progress  on  the  outstanding  FY06  recommendation 
to  include  cyber  assessments  during  BMDS-centric  C2BMC 
testing.  A  pathfinder  exercise  is  planned  during  the  Ground 
Test-06  campaign  in  FY15-16  with  a  more  complete  cyber 


assessment  during  the  Ground  Test-07  campaign  in  FY17-18. 
The  MDA  is  relying  on  C2BMC  S8.2  (FY17-18)  to  address  the 
remaining  FY13  recommendation  on  C2BMC  interoperability 
and  engagement  management  deficiencies  uncovered  during 
numerous  tests,  including  most  recently  FTO-Ol. 

•  FY 14  Recommendations.  None. 
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Ground-Based  Midcourse  Defense  (GMD) 


Executive  Summary 

•  Ground-based  Midcourse  Defense  (GMD)  has  demonstrated 
a  limited  capability  to  defend  the  U.S.  Homeland  from  small 
numbers  of  intermediate-range  or  intercontinental  ballistic 
missile  threats  launched  from  North  Korea  or  Iran. 

•  In  a  developmental  test  environment,  the  Missile  Defense 
Agency  (MDA)  conducted  the  first  successful  intercept  by 
the  Capability  Enhancement  II  (CE-II)  exoatmospheric  kill 
vehicle  (EKV)  since  the  failure  of  the  CE-II  vehicle  during 
intercept  flight  tests  in  FY 10  and  FY 1 1 . 

•  The  MDA  Director  commissioned  an  Independent  Expert 
Panel,  which  completed  a  comprehensive  assessment  of 
the  Ground-Based  Interceptor  (GBI)  program  to  identify 
deficiencies  that  could  preclude  predictable,  reliable  GBI 
operations. 

System 

GMD  is  a  Ballistic  Missile  Defense  System  (BMDS)  element 

that  counters  intermediate-range  and  intercontinental  ballistic 

missile  threats  to  the  U.S.  Homeland.  The  GMD  consists  of  and 

operates: 

•  GBls  at  Fort  Greely,  Alaska,  and  Vandenberg  AFB,  California 

•  GMD  ground  system  including  GMD  Fire  Control  (GFC) 
nodes  at  Schriever  AFB,  Colorado,  and  Fort  Greely, 

Alaska;  Command  Launch  Equipment  at  Vandenberg  AFB, 
California,  and  Fort  Greely,  Alaska;  and  In-Flight  Interceptor 
Communication  System  Data  Terminals  at  Vandenberg  AFB, 
California;  Fort  Greely,  Alaska;  and  Eareckson  Air  Station, 
Alaska 

•  GMD  secure  data  and  voice  communication  system 
including  long-haul  communications  using  the  Defense 
Satellite  Communication  System,  commercial  satellite 
communications,  and  fiber-optic  cable  (both  terrestrial  and 
submarine) 

•  COBRA  DANE  Upgraded  Radar  at  Eareckson  Air  Station 
(Shemya  Island),  Alaska 

•  Upgraded  Early  Warning  Radars  at  Beale  AFB,  California; 
Royal  Air  Force  Fylingdales,  United  Kingdom;  and  Thule  Air 
Base,  Greenland 

•  External  interfaces  that  connect  to  Aegis  BMD;  North 
American  Aerospace  Defense/U.S.  Northern  Command 
Command  Center;  and  the  Command  and  Control,  Battle 
Management,  and  Communications  system  at  Peterson  AFB, 
Colorado;  Space-Based  Infrared  System/Defense  Support 
Program  (SBIRS/DSP)  at  Buckley  AFB,  Colorado;  AN/TPY-2 


(Forward-Based  Mode)  radar  at  Shariki  Air  Base,  Japan;  and 
the  Sea-Based  X-band  (SBX)  radar,  a  sea-based  mobile  sensor 
platform 

Mission 

Military  operators  for  the  U.S.  Army  Space  and  Missile  Defense 
Command/ Army  Forces  Strategic  Command  (the  Army  service 
component  to  U.S.  Strategic  Command)  will  use  the  GMD 
system  to  defend  the  U.S.  Homeland  against  intermediate-range 
and  intercontinental  ballistic  missile  attacks  using  the  GBI  to 
defeat  threat  missiles  during  the  midcourse  segment  of  flight. 

Major  Contractors 

•  The  Boeing  Company,  Network  and  Space 
Systems  -  Huntsville,  Alabama 

•  Orbital  Sciences  Corporation,  Missile  Defense 
Systems  -  Chandler,  Arizona 

•  Raytheon  Company,  Missile  Systems  -  Tucson,  Arizona 

•  Northrop  Grumman  Corporation,  Information 
Systems  -  Huntsville,  Alabama 
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Activity 

•  The  MDA  conducted  testing  in  accordance  with  the 
DOT&E- approved  Integrated  Master  Test  Plan. 

•  The  MDA  conducted  Ground  Test  lntegrated-04e  (GTI-04e) 
Part  2  in  April  through  May  2014.  The  MDA  used  hardware 
and  software  representations  of  the  GMD;  the  SB1RS/DSP; 
the  Command  and  Control,  Battle  Management,  and 
Communications;  the  AN/TPY-2  (Forward-Based  Mode) 
radar;  the  Aegis  BMD  radar  in  its  long-range  surveillance 
and  track  capability;  the  Upgraded  Early  Warning  Radars; 
and  the  SBX  radar  to  investigate  GMD  capabilities  against 
intercontinental  ballistic  missile  threats. 

•  The  MDA  executed  an  intercept  flight  test  of  a  GBI  equipped 
with  a  CE-II  EKV  against  an  intermediate-range  ballistic 
missile  target  in  June  2014.  During  this  developmental  test, 
Flight  Test  GBI-06b  (FTG-06b),  the  MDA  demonstrated 
fixes  made  to  the  CE-II  EKV  subsequent  to  the  FTG-06a 
Failure  Review  Board  findings.  They  also  demonstrated 
GBI  performance  under  more  challenging  threat  engagement 
conditions  than  previous  intercept  flight  tests. 

-  The  MDA  launched  an  intermediate-range  ballistic  missile 
target  from  the  U.S.  Army’s  Reagan  Test  Site  on  Kwajalein 
Atoll,  Republic  of  the  Marshall  Islands. 

-  The  Aegis  BMD  radar  performed  long-range  surveillance 
and  track.  The  SBX  radar  performed  long-range  detection, 
tracking,  and  target  discrimination.  The  SBIRS/DSP  also 
participated  in  this  test. 

-  The  GFC  planned  the  intercept  and  a  GFC  military 
operator  directed  launch  of  the  test  GBI  from  Vandenberg 
AFB,  California. 

•  The  MDA  Director  commissioned  an  Independent  Expert 
Panel  in  August  2013  to  characterize  the  deployed  GBI 


fleet.  Their  goal  was  to  identify  design,  manufacturing,  and 
quality  deficiencies  that  would  negatively  affect  predictable 
and  reliable  GBI  operations.  The  panel  completed  their 
assessment  in  March  2014. 

Assessment 

•  GTl-04e  Part  2  demonstrated  upgraded  GMD  software  builds 
with  other  BMDS  elements  and  the  SBIRS/DSP.  The  MDA 
identified  limitations  and  is  continuing  to  investigate. 

•  During  FTG-06b,  the  MDA  demonstrated  a  long  interceptor 
time-of-flight,  medium-closing  velocity  engagement  of  an 
intermediate -range  ballistic  missile  by  the  CE-II  GBI.  The 
system  performed  all  EKV  functions  to  discriminate  and 
intercept  a  lethal  object  from  a  representative  intercontinental 
ballistic  missile  target  scene  with  countermeasures.  All  key 
performance  parameters  were  within  prescribed  constraints 
and  pre-mission  predictions.  The  intercept  was  confirmed  by 
multiple  data  sources. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  MDA  has 
addressed  previous  recommendations  with  the  exception  of 
the  FY13  recommendation  to  retest  the  CE-I  EKV  in  order  to 
accomplish  the  test  objectives  from  the  failed  FTG-07  mission. 
The  first  opportunity  to  address  this  recommendation  will  be 
4QFY17. 

•  FY 14  Recommendation. 

1.  The  MDA  should  extend  the  principles  and 

recommendations  contained  in  the  Independent  Expert 
Panel  assessment  report  on  the  GBI  fleet  to  all  components 
of  the  BMDS  instantiation  for  Homeland  Defense. 
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Sensors 


Executive  Summary 

•  The  Missile  Defense  Agency  (MDA)  has  gained  significant 
operational  experience  with  each  of  the  Ballistic  Missile 
Defense  System  (BMDS)  sensors  since  the  completion  of 
sensor  upgrade  and  development  programs. 

•  During  FY14,  BMDS  sensors  participated  in  two  major 
ground  tests,  three  flight  tests,  and  observed  numerous  targets 
of  opportunity  such  as  domestic  or  foreign  launch  events. 

•  Accreditation  of  each  of  the  sensor  models  for  use  in 
performance  assessments  continues  to  progress,  but  is  still 
incomplete. 

System 

The  BMDS  sensors  are  systems  that  provide  real-time  ballistic 
missile  threat  data  to  the  BMDS.  The  data  are  used  to  counter 
ballistic  missile  attacks.  These  sensor  systems  are  operated 
by  die  Army,  Navy.  Ah'  Force,  and  the  MDA,  and  include  a 
satellite-based,  infrared  sensor  system  and  five  phased  array  radar- 
system  types.  The  sensor  systems  are: 

•  Space-Based  Infrared 
System/Defense  Support 
Program  (SBIRS/DSP).  a 
satellite  constellation  of 
infrared  sensors  operated 
by  the  Air  Force  with  an 
external  interface  to  the 
BMDS  located  at  Buckley 
AFB,  Colorado 

•  Fixed  site,  fixed  orientation, 
phased  array  radars 

-  COBRA  DANE  Radar 
Upgrade  (CDU).  an 
L-band  radar  (one 
radar  face  that  provides 
120-degree  azimuth 
field  of  view-)  operated 
by  the  Air  Force  and 
located  at  Eareckson  Air 
Station  (Shemya  Island). 

Alaska. 

-  Upgraded  Early 
Warning  Radars 
(UEWRs),  ultra-high 
frequency  radars 
operated  by  the  Air 
Force  and  located  at 
Beale  AFB.  California 
(two  radar  faces  that 
provide  240-degree 
azimuth  field  of  view);  Fylingdales,  United  Kingdom 
(three  radar  faces  that  provide  360-degree  azimuth  field  of 


COBRA  DANE 


view);  and  Thule.  Greenland  (tw-o  radar  faces  that  provide 
240-degree  azimuth  field  of  view).  The  MDA  and  Air 
Force  Space  Command  (AFSPC)  awrarded  a  contract  in 
July  2012  for  the  upgrade  of  the  Early  Warning  Radar 
(EWR)  at  Clear  Air  Force  Station,  Alaska.  In  December 
2012,  a  contract  option  was  exercised  for  the  upgrade  of 
the  EWR  at  Cape  Cod  Air  Force  Station,  Massachusetts. 

•  Mobile/transportable  phased  array  radars 


-  AN/TPY-2 

Forw  ard-Based  Mode 
(FBM)  radars,  X-band 
radars  operated  by  the 
Army  and  located  at 
sites  in  Japan,  Israel, 
Turkey,  and  the  U.S. 
Central  Command  area 
of  responsibility. 

-  AN/SPY- 1  radars  in 
track  and  cue  mode, 
S-band  radars  (frmr 
radar  faces  that 
provide  360-degree 
azimuth  field  of  view) 
operated  by  the  Navy 
and  located  aboard  33 
Aegis  Ballistic  Missile 
Defense  (BMD)- 
capable  cruisers  and 
destroyers. 

-  SBX  radar,  an  X-band 
radar  operated  by  the 
MDA  and  located 
aboard  a  twin-hulled, 
semi-submersible, 
self-propelled, 
ocean-  going  platform. 


i 


Aegis  BMD 


?  ’ 


** 


i 


Mission 

Military  operators  for  the  U.S.  Strategic  Command.  U.S. 

Northern  Command,  U.S.  European  Command.  U.S.  Pacific 

Command,  and  U.S.  Central  Command  will  use  the  BMDS 

sensors  to: 

•  Detect,  track,  and  classify  ballistic  missile  threats  that  target 
the  United  States  and  U.S.  allies 

•  Provide  data  for  situational  awareness  and  battle  management 
to  the  Combatant  Commands  through  the  BMDS  Command 
and  Control.  Battle  Management,  and  Communications 
(C2BMC) 

•  Provide  data  that  support  engagement  of  ballistic  missile 
threats  by  ballistic  missile  defense  systems 


Sensors 
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Major  Contractors 

•  AN/SPY- 1  Radar  (Aegis  BMD):  Lockheed 
Martin  Corporation,  Mission  Systems  and 
Training  -  Moorestown,  New  Jersey 

•  AN/TPY-2  (FBM)  Radar,  CDU  Radar,  SBX  Radar,  and 
UEWRs/EWRs:  Raytheon  Company,  Integrated  Defense 
Systems  -  Tewksbury,  Massachusetts 


•  SB1RS:  Lockheed  Martin  Space  Systems  -  Sunnyvale, 
California 

•  DSP:  Northrop  Grumman  Corporation,  Aerospace 
Systems  -  Redondo  Beach,  California 


Activity 

The  MDA  conducted  testing  during  FY14  in  accordance  with  the 
DOT&E-approved  Integrated  Master  Test  Plan. 

Aegis  BMD  Radar 

•  The  MDA  used  hardware  and  software  representations 
of  the  Aegis  BMD  radar  in  its  hardware-in-the-loop 
(HWIL)  test,  Ground  Test  Integrated-04e  (GTI-04e) 

Part  2,  in  April  and  May  2014.  In  this  test,  the  Aegis 
BMD  radar  representation  detected  and  tracked  simulated 
intercontinental  ballistic  missile  (ICBM)  threats  to  the 
U.S.  Homeland  and  forwarded  the  track  data  to  an  HWIL 
representation  of  the  C2BMC. 

•  The  Aegis  BMD  radar,  in  its  long-range  surveillance  and 
track  mode,  participated  in  Flight  Test  Ground-Based 
Interceptor-06b  (FTG-06b)  in  June  2014.  The  Aegis  BMD 
radar  detected  and  tracked  the  intermediate-range  ballistic 
missile  (1RBM)  target  and  forwarded  the  track  data  to  the 
C2BMC  system. 

AN/TPY-2  (FBM)  Radar 

•  The  MDA  used  HWIL  representations  of  the  AN/TPY-2 
(FBM)  radar  in  GTl-04e  Part  2  in  April  and  May  2014. 

The  AN/TPY-2  (FBM)  radar  representations  detected  and 
tracked  simulated  ICBM  threats  to  the  U.S.  Homeland  and 
forwarded  the  track  data  to  an  HWIL  representation  of  the 
C2BMC. 

•  The  MDA  used  HWIL  representations  of  the  AN/TPY-2 
(FBM)  radar  in  its  Fast  Exchange  HWIL  ground  test  in 
June  2014.  The  MDA  used  three  AN/TPY-2  (FBM)  radars 
in  its  Fast  Exchange  distributed  ground  test  in  August  2014. 
In  the  Fast  Exchange  ground  tests,  the  MDA  investigated 
the  level  of  mutual  sensor  support  between  multiple 
AN/TPY-2  (FBM)  radars  and  the  C2BMC  system. 

CDU  Upgrade 

•  In  FY14,  the  U.S.  Air  Force  used  the  CDU  radar  to  observe 
targets  of  opportunity.  The  AFSPC  also  used  the  CDU 
radar  as  a  contributory  sensor  to  the  Space  Surveillance 
Network  to  track  orbital  debris  and  active  satellites. 

SBIRS/DSP 

•  In  FY14,  the  U.S.  Air  Force  used  the  SBIRS/DSP  system 
to  observe  domestic  and  foreign  launch  events  and 
provide  launch  event  data  to  the  operational  BMDS. 

The  SBIRS/DSP  system  also  participated  in  Flight  Test 
Operational-01,  Flight  Test  Standard  Missile-22,  and 
FTG-06b. 

•  A  digital  representation  of  the  SBIRS/DSP  system 
participated  in  GTI-04e  Part  2  in  April  and  May  2014. 


SBX  Radar 

•  In  FY14,  the  SBX  began  in  limited  test  support  status  but 
now  serves  as  both  a  test  and  operational  asset.  The  SBX 
radar  can  be  deployed  based  on  warning  of  an  ICBM  threat 
to  the  U.S.  Homeland  and  for  BMDS  flight  testing. 

•  A  HWIL  representation  of  the  SBX  radar  participated 
in  GTI-04e  Part  2  in  April  and  May  2014.  In  GTI-04e 
Part  2,  the  SBX  radar  representation  detected,  tracked, 
and  discriminated  simulated  ICBM  threats  to  the  U.S. 
Homeland  and  forwarded  the  track  data  to  an  HWIL 
representation  of  the  Ground  Based  Midcourse  Defense 
(GMD)  Fire  Control  (GFC). 

•  The  SBX  radar  participated  in  FTG-06b  in  June  2014. 

The  SBX  radar  accepted  sensor  task  plans  from  the  GFC, 
detected,  tracked,  and  discriminated  the  1RBM  target,  and 
forwarded  track  data  to  the  GFC. 

UEWRs/EWRs 

•  In  FY14,  the  MDA  transferred  the  sustainment 
responsibility  for  the  Beale,  Fylingdales,  and  Thule  UEWRs 
to  AFSPC. 

•  In  FY14,  the  U.S.  Air  Force  used  the  Beale,  Fylingdales, 
and  Thule  UEWRs,  and  the  Clear  and  Cape  Cod  EWRs, 
to  observe  targets  of  opportunity.  The  AFSPC  also  used 
these  radars  as  collateral  sensors  to  the  Space  Surveillance 
Network  to  track  orbital  debris  and  active  satellites. 

Assessment 

The  MDA  has  gained  significant  operational  experience  with 
each  of  the  BMDS  sensors  since  the  completion  of  the  sensor 
upgrade  and  development  programs.  The  MDA  and  the  BMDS 
Operational  Test  Agency,  however,  have  not  fully  accredited 
models  and  simulations  of  the  BMDS  sensors  for  performance 
assessment.  Representations  of  the  AN/TPY-2  (FBM)  radar, 
the  SBX  radar,  the  UEWR,  the  Aegis  BMD  radar,  and  the 
SBIRS/DSP  system  have  been  accredited  for  limited  uses. 
Representations  of  the  CDU  radar  have  not  been  accredited. 

Aegis  BMD  Radar 

•  In  GTl-04e  Part  2,  the  MDA  demonstrated  a  capability  of  the 
Aegis  BMD  radar  to  support  GMD  engagement  of  IRBM 
and  ICBM  threats.  The  Aegis  BMD  radar  provided  data 
that  supported  BMDS  situational  awareness,  BMDS  sensor 
tasking,  and  GMD  engagement  planning. 

•  During  FTG-06b,  Aegis  BMD  acquired  target  objects  using 
its  pre-defined  autonomous  search  plan,  generated  tracks 
of  the  target  objects,  and  reported  two  clusters  of  objects 
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to  the  C2BMC.  Based  on  pre -mission  analysis,  the  MDA 
anticipated  only  one  cluster  of  objects  being  detected  and  is 
investigating  why  this  second  cluster  was  observed.  Although 
not  experienced  in  this  test  and  under  different  circumstances, 
this  second  cluster  of  objects  could  have  generated  a  second 
engagement,  potentially  resulting  in  Ground-based  Interceptor 
wastage. 

AN/TPY-2  (FBM)  Radar 

•  In  GTl-04e  Part  2,  the  MDA  demonstrated  a  capability  of 
the  AN/TPY-2  (FBM)  radar  to  support  GMD  engagement 
of  1RBM  and  ICBM  threats.  The  AN/TPY-2  (FBM)  radar 
representations  provided  data  on  the  simulated  missile 
threats  to  the  C2BMC  system  that  supported  BMDS 
situational  awareness,  BMDS  sensor  tasking,  and  GMD 
engagement  planning. 

•  In  the  Fast  Exchange  ground  tests,  the  MDA  proved 
the  concept  of  multiple  AN/TPY-2  (FBM)  radars  being 
mutually  supportive  across  Combatant  Command’s  areas 
of  responsibility.  The  AN/TPY-2  (FBM)  radars  detected 
and  tracked  simulated  missile  threats  to  multiple  locations, 
forwarded  the  track  data  to  the  C2BMC  system,  and 
received  and  responded  to  C2BMC  system  tasking. 

CDU  Upgrade 

•  Due  to  its  location  and  field  of  view,  the  CDU  radar  has  not 
participated  in  BMDS  intercept  flight  tests.  Flowever,  the 
CDU  radar  did  observe  numerous  targets  of  opportunity. 

SBIRS/DSP 

•  SBIRS/DSP  performance  and  its  capability  to  support 
BMDS  engagement  of  IRBM  and  ICBM  threats  are 
classified.  Detailed  analysis  can  be  found  in  DOT&E’s 
February  2014  BMDS  Annual  Report. 

SBX  Radar 

•  In  GTI-04e  Part  2,  the  MDA  demonstrated  the  capability  of 
the  SBX  radar  to  support  GMD  engagement  of  IRBM  and 


ICBM  threats.  After  detecting,  tracking,  and  discriminating 
the  inbound  missiles,  the  SBX  radar  transmitted 
engagement- quality  data  to  the  GFC  for  intercept 
prosecution. 

•  The  SBX  radar  performed  nominally  during  the  FTG-06b 
mission.  It  detected  and  tracked  the  target  missile, 
accurately  discriminated  all  target  objects,  and  transmitted 
and  received  target  information  with  the  other  elements  in 
a  timely  manner.  It  correctly  determined  the  engagement 
hit  assessment  and  reported  such  to  the  GMD  fire  control 
component. 

UEWRs/EWRs 

•  Due  to  their  locations  and  fields  of  view,  the  UEWRs  at 
Thule  and  Fylingdales  have  not  participated  in  BMDS 
intercept  flight  tests  in  an  operationally  realistic  manner. 
Beale  has  participated  in  all  flight  tests  within  its  field  of 
view  and  has  supplied  critical  data  in  analysis  of  these  flight 
tests.  Data  from  targets  of  opportunity  and  ground  tests 
support  performance  estimates  for  the  current  configuration 
of  the  UEWRs. 

•  The  MDA  and  the  U.S.  Air  Force  have  not  yet  upgraded  the 
EWRs  at  Clear  and  Cape  Cod  Air  Force  Stations,  and  these 
radars  are  not  yet  part  of  the  MDA’s  sensor  network. 

Recommendations 

•  Status  of  Previous  Recommendations.  The  MDA 
addressed  all  but  two  previous  recommendations.  The 
FY09  recommendation  on  developing  sensor  concepts  of 
operations  in  conjunction  with  the  Combatant  Commands 
and  demonstrating  them  in  operational  testing  has  yet  to  be 
completed.  The  FY12  recommendation  to  conduct  AN/TPY-2 
(FBM)  cyber  testing  also  remains  open. 

•  FY 14  Recommendations.  None. 
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Terminal  High-Altitude  Area  Defense  (THAAD) 


Executive  Summary 

•  The  Terminal  High- Altitude  Area  Defense  (THAAD)  Project 
Office  performed  many  obsolescence  redesigns,  hardware 
and  software  upgrades,  and  corrective  actions  during  FY14. 
Additionally,  the  Missile  Defense  Agency  (MDA)  advanced 
the  characterization  of  its  associated  AN/TPY-2  radar  debris 
mitigation  algorithms.  The  MDA  should  plan  to  rigorously 
test  the  THAAD  Configuration  2  system  in  order  to  assess 
these  changes. 

•  The  THAAD  Project  Office  has  closed  18  of  the  39 
Conditional  Materiel  Release  conditions  imposed  on  the  first 
two  THAAD  batteries.  Fixes  and  testing  of  the  remaining 
conditions  are  scheduled  through  FY17. 

•  THAAD  system  reliability  and  maintainability  measures 
have  improved,  but  still  vary  greatly  from  event  to  event. 
Additional  testing,  planned  for  FY15,  is  required  to  determine 
if  overall  system  reliability  is  maturing. 

System 

•  The  THAAD  ballistic  missile  defense  system  consists  of  five 
major  components: 

-  Missiles 

-  Launchers 

-  Radar  (designated  AN/TPY-2  Terminal  Mode  [TM]) 

-  THAAD  Fire  Control  and  Communications 

-  THAAD  Peculiar  Support  Equipment 

•  THAAD  can  accept  target  cues  for  acquisition  from  Aegis 
Ballistic  Missile  Defense,  satellites,  and  other  external  theater 
sensors  and  command  and  control  systems. 

•  THAAD  is  intended  to  complement  the  lower-tier  Patriot 
system  and  the  upper-tier  Aegis  Ballistic  Missile  Defense 
system. 

Mission 

U.S.  Strategic  Command  intends  to  deploy  THAAD  to  protect 
critical  assets  worldwide.  Commanders  will  use  THAAD  to 


intercept  incoming  short-  to  intermediate-range  ballistic  missile 
threats  in  the  endo-  or  exoatmosphere. 

Major  Contractors 

•  Prime:  Lockheed  Martin  Corporation,  Missiles  and  Fire 
Control  -  Dallas,  Texas 

•  AN/TPY-2  Radar  (TM):  Raytheon  Company,  Integrated 
Defense  Systems  -  Tewksbury,  Massachusetts 


Activity 

•  The  Flight  Test  Operational-Ol  (FTO-Ol)  Ballistic  Missile 
Defense  System  (BMDS)-level  flight  test  in  September  2013 
included  two  THAAD  engagements. 

•  No  additional  THAAD  flight  testing  occurred  during  FY14, 
which  was  in  accordance  with  the  DOT&E-approved 
Integrated  Master  Test  Plan.  However,  the  THAAD  Project 
Office  performed  many  obsolescence  redesigns,  software  and 
hardware  upgrades,  and  corrective  actions  throughout  FY14, 
which  could  contribute  to  significant  capability  changes  in 
FY15. 


•  During  FY14,  THAAD  participated  in  numerous  BMDS-level 
ground  tests:  Ground  Test  lntegrated-04e  Part  la  and  Part  2, 
Fast  Phoenix,  and  Fast  Exchange. 

•  The  THAAD  program  conducted  qualification  testing  and 
demonstrations  of  THAAD  training  devices,  and  system 
integration  and  checkout  of  THAAD  Configuration  2. 

•  The  THAAD  program  conducted  planning  for  three  major 
test  events  scheduled  for  FY15:  FTO-02  Event  2,  Flight  Test 
THAAD- 18,  and  the  Reliability  Growth  Test. 
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•  The  Army  reviewed  and  assessed  reliability  and 
maintainability  data  from  the  FTO-Ol  flight  test. 

Assessment 

•  The  classified  DOT&E  February  2014  BMDS  Annual  Report, 
Appendix  E,  assesses  the  results  from  the  FTO-Ol  mission  and 
includes  six  key  findings  covering  THAAD  participation  and 
system-level  performance,  effectiveness,  and  suitability. 

•  During  ground  tests  in  FY14,  THAAD  exercised  debris 
mitigation  algorithms,  advancing  the  characterization  of 
this  key  component  of  THAAD  Configuration  2  capability. 
However,  flight  testing  will  be  required  to  complete  the 
characterization. 

•  The  THAAD  program  continued  efforts  to  achieve  a  Full 
Materiel  Release  of  the  first  two  THAAD  batteries,  which 
achieved  Conditional  Materiel  Release  in  February  2012.  The 
THAAD  Project  Office  continued  to  address  the  39  conditions 
that  need  to  be  resolved  before  the  Army  could  grant  a  Full 
Materiel  Release.  Fixes  and  testing  of  the  21  open  conditions 
are  scheduled  through  FY17.  In  addition  to  the  7  conditions 
closed  in  FY12  and  FY13,  11  were  closed  in  FY14: 

-  Verification  of  technical  manuals 

-  Hazard  classification  for  missile  transport 

-  A  safety  review  of  the  technical  manuals 

-  Verifying  missile  non-operating  performance  requirements 
in  cold  environments 

-  Verifying  the  missile  Thermally  Initiated  Venting  System  in 
a  flight  test 

-  Correcting  common  datalink  messaging  issues  for 
Precision  Position  Location  Information  message 
generation  and  receipt 

-  Changing  contingent  lethal  object  logic  in  the  fire  control 

-  Safety  verification  of  insensitive  munitions  requirements 

-  Implementing  configuration  control  for  software  and 
firmware  management 

-  Resolving  radar  software  message  problems  with  fixes 
installed  in  fielded  batteries 

-  Completing  missile  field  and  storage  maintenance  and 
inspection  procedures 


•  Comparing  the  reliability  and  maintainability  data  from 
FTO-Ol  to  the  previous  results  from  Flight  Test  Integrated-01 
(1QFY13),  Flight  Test  THAAD- 12  (1QFY12),  and  the 
Reliability  Confidence  Test  (4QFY11)  shows  that  reliability 
and  maintainability  measures  are  still  fluctuating  greatly 
between  test  events.  FTO-Ol  results  were  generally  improved 
from  Flight  Test  Integrated  01,  but  given  the  differences 
between  events,  data  from  a  longer,  operationally  realistic  test 
event  such  as  the  upcoming  Reliability  Growth  Test  in  FY15 
are  needed  to  determine  if  the  overall  system  reliability  is 
maturing. 

Recommendations 

•  Status  of  Previous  Recommendations. 

-  The  classified  DOT&E  February  2012  THAAD  and 
AN/TPY-2  Radar  Operational  and  LFT&E  Report 
contained  7  recommendations  in  addition  to  and  not 
associated  with  the  original  39  Conditional  Materiel 
Release  conditions  established.  Two  recommendations 
have  been  addressed.  The  MDA  should  continue  to 
address  the  remaining  three  classified  recommendations 
(Effectiveness  #2,  Effectiveness  #5,  and  Survivability  #4). 
In  addition,  the  MDA  and  the  Army  should  conduct 
electronic  warfare  testing  and  analysis  (Suitability  #3) 
and  implement  equipment  redesigns  and  modifications 
identified  during  natural  environment  testing 
(Suitability  #11). 

-  DOT&E  made  one  additional  recommendation  in  FY13  for 
the  THAAD  Project  Office  to  reassess  their  reliability  and 
maintainability  growth  planning  curve.  Progress  was  made 
on  this  recommendation;  however,  it  was  hampered  by 
THAAD  component  configurations  remaining  in  flux. 

•  FY 14  Recommendation. 

1 .  The  MDA  should  plan  to  rigorously  test  the  THAAD 
Configuration  2  system  in  order  to  assess  obsolescence  and 
other  redesigns  of  hardware  and  software. 
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Live  Fire  Test  and  Evaluation  (LFT&E) 


DOT&E  executed  oversight  of  survivability  and  lethality  test  and 
evaluation  for  121  acquisition  programs  in  FY14.  Of  those  121 
programs,  21  operated  under  the  waiver  provision  of  U.S.  Code, 
Title  10,  Section  2366,  by  executing  an  approved  alternative 
Live  Fire  Test  and  Evaluation  (LFT&E)  strategy  in  lieu  of  full-up 
system-level  testing.  DOT&E  submitted  eight  reports  on  LFT&E 
results  prior  to  programs  entering  into  full-rate  production. 

DOT&E  published  reports  on  the  following  programs  during  the 
past  year  (reports  marked  with  an  asterisk  were  sent  to  Congress): 

LFT&E  Reports 

•  F1ELLFIRE  Romeo  Final  Lethality  Assessment 

•  HELLFIRE  R-9E  Initial  Lethality  Assessment 

•  Modernized  Expanded  Capacity  Vehicle-Survivability 
(MECV-S)  Survivability  Assessment  Report* 

DOT&E  Reports  (with  combined  OT&E/LFT&E  results) 

•  Joint  High  Speed  Vessel  (JHSV)  Initial  Operational  Test  and 
Evaluation  (IOT&E)  and  LFT&E  Report* 

•  M982E1  Excalibur  Increment  lb  IOT&E  and  LFT&E  Report* 

•  P-8  Poseidon  Multi-Mission  Maritime  Aircraft  (MMA) 

IOT&E  Report* 

•  Littoral  Combat  Ship  (LCS)  Early  Fielding  Report* 


Special  Reports 

•  DOT&E  Response  to  Chief  of  Naval  Operations  Report 
to  Congress  on  the  Current  Concept  of  Operations  and 
Expected  Survivability  Attributes  of  the  Littoral  Combat 
Ship* 

In  addition  to  satisfying  acquisition  oversight  requirements,  the 

LFT&E  program: 

•  Funds  and  executes  technical  oversight  on  investment 
programs  that  provide  joint  munitions  effectiveness 
data  (Joint  Technical  Coordinating  Group  for  Munitions 
Effectiveness) 

•  Funds  projects  to  develop  advanced  technologies  and 
analytical  methods  to  increase  aircraft  survivability  (Joint 
Aircraft  Survivability  Program) 

•  Conducts  vulnerability  and  lethality  testing  of  fielded 
platforms  and  weapons  systems  and  improves  survivability 
analysis  tools  (Joint  Live  Fire  Program) 

•  Supports  quick  reaction  efforts  addressing  urgent  operational 
commander’s  needs. 


JOINT  TECHNICAL  COORDINATING  GROUP  FOR  MUNITIONS  EFFECTIVENESS  (JTCG/ME) 


Joint  Logistics  Commanders  chartered  the  Joint  Technical 
Coordinating  Group  for  Munitions  Effectiveness  (JTCG/ME) 
more  than  40  years  ago  to  serve  as  the  DOD’s  focal  point  for 
munitions  effectiveness  information.  This  has  taken  the  form  of 
widely  used  Joint  Munitions  Effectiveness  Manuals  (JMEMs), 
which  address  all  major,  non-nuclear  U.S.  weapons.  JTCG/ME 
authenticates  weapons  effectiveness  data  for  use  in  operational 
weaponeering,  strike  mission  planning,  training,  systems 
acquisition,  weapon  procurement,  and  combat  modeling  and 
simulation.  The  Armed  Forces  of  the  U.S.,  NATO,  and  other 
allies  use  JMEMs  to  plan  operational  missions,  support  training 
and  tactics  development,  and  support  force-level  analyses. 
JTCG/ME  also  develops  and  standardizes  methods  for 
evaluation  of  munitions  effectiveness  and  maintains  databases 
for  target  vulnerability,  munitions  lethality,  and  weapon  system 
accuracy.  The  JMEM  requirements  and  development  processes 
continue  to  be  driven  by  operational  lessons  learned  and  the 
needs  of  Combatant  Commands,  Services,  Military  Targeting 
Committees,  and  Operational  Users  Working  Groups  for 
specific  weapon-target  pairings  and  methodologies. 

The  primary  JMEM  application  is  weaponeering,  which  is 
the  detailed  technical  planning  of  a  combat  strike.  Strike 
planning  occurs  at  multiple  levels  in  the  operational  chain  of 


command  before  actual  combat.  JTCG/ME  produces,  distributes, 
and  regularly  updates  JMEMs,  which  provide  computerized 
effectiveness  tools  and  data  for  rapid  evaluation  of  alternative 
weapons  and  their  delivery  against  specific  targets.  In  many 
cases,  effectiveness  and  collateral  damage  estimates  generated  by 
these  tools  are  part  of  the  decision  criteria  for  strikes  approved  at 
the  highest  levels  of  the  U.S.  Government. 

JOINT  MUNITIONS  EFFECTIVENESS  MANUAL  (JMEM) 
TARGETING  AND  WEAPONEERING  SOFTWARE 

U.S.  Africa  Command  (USAFRICOM)  and  U.S.  Central 
Command  (USCENTCOM)  used  JTCG/ME  Joint  Munitions 
Effectiveness  Manual  Weaponeering  System  (JWS)  v2.1 
software  and  Collateral  Effects  Radii  tables  for  operational 
weaponeering  and  collateral  damage  estimation  in  direct  support 
of  operations  in  the  USAFRICOM  and  USCENTCOM  Areas  of 
Responsibilities.  The  JTCG/ME  developed  various  analytical 
and  operational  methodologies  and  target  models  to  provide 
continued  support  to  operational  commanders,  weaponeers, 
and  planners.  Additionally,  JTCG/ME  released  the  air-to-air 
and  surface-to-air  planning  model,  the  Joint  Anti-air  Combat 
Effectiveness  System  (J-ACE)  v5.2.1,  in  April  2014  to  provide 
aircraft  survivability  data. 


LFT&E  319 


F  Y 1 4  LFT&E  PROGRAM 


The  fielded  JWS  v2.1  software  incorporates  integral  building 
analysis  and  hardened  target  modules  to  create  a  merged  tool 
that  generates  weapon  effectiveness  and  damage  assessments 
against  infrastructure  targets  to  include  buildings,  bunkers,  and 
tunnels.  JWS  v2.1  contains  new/updated  targets  and  munitions, 
new  explosive  equivalent  weights  based  on  blast  testing,  as  well 
as  an  improved  3-D  viewer.  JWS  v2.2  development  is  ongoing  to 
support  coalition  partners  for  fielding  in  FY15. 

J-ACE  v5.2.1  simulates  air-to-air  and  surface-to-air  engagements 
and  includes  Blue,  Red,  and  Gray  air-to-air  missile  and  Red 
and  Gray  surface-to-air  missile  fly-out  models.  J-ACE  v5.2.1 
provides  an  updated  Joint  Anti-Air  Model  (JAAM),  a  missile 
fly-out  model  including  hundreds  of  weapon  target  pairings 
and  JAAM -Enhanced  surface-to-air  missile  countermeasures 
simulation.  J-ACE  v5.2.1  also  provides  new  Endgame  Manager 
(EM)  software  and  data  sets,  which  add  missile  lethality  and 
target  vulnerability.  EM  allows  explicit  evaluation  of  weapon 
miss  distance,  fuze  performance,  weapon  lethality,  and  target 
vulnerability.  EM  provides  a  Probability  of  Kill  given  an 
intercept.  In  addition,  J-ACE  v5.3  is  being  developed  to 
provide  extended  and  updated  data  sets  for  missile  and  aircraft 
target  aero-perfonnance,  anti-air  missile  lethality,  and  air  target 
vulnerability. 

OPERATIONAL  SUPPORTTO  MISSION  PLANNING 

The  JTCG/ME  provided  updates  for  Collateral  Effects  Radii 
values  for  newly-fielded/updated  weapons  (e.g.,  Small  Diameter 
Bomb  (SDB)  II,  Griffin,  HELLFIRE,  GBU-49,  BLU-133, 
etc.),  in  support  of  the  Combatant  Commands.  In  addition,  the 
JTCG/ME  released  the  Digital  Precision  Strike  Suite  Collateral 
Damage  Estimation  (DCiDE)  vl.1.1  tool,  which  has  the 
capability  to  calculate  collateral  damage  estimates  along  a  route. 
This  new  capability  has  been  used  in  support  of  multiple  kinetic 
strikes  by  the  task  force  in  Afghanistan.  Additionally,  JTCG/ME 
trained  nearly  300  users  at  12  different  commands  to  support 
Collateral  Damage  Estimation  decisions. 


JWS  v2.1  has  ongoing  initiatives  to  include  a  new  imagery 
interface  to  implement  aimpoint  development  that  leverages 
the  Tasked  Target  Text  Data  (T3D)  format  implemented  by 
currently  fielded  mission  planning  systems.  JWS  software 
and  the  T3D  imagery  interface  will  be  modified  to  support 
integration  of  electronic  light  table  viewers.  Also,  Modernized 
Integrated  Database  and  Joint  Targeting  Toolbox  interfaces  will 
be  developed  with  additional  capabilities  to  support  connectivity. 
These  developments  will  enable  the  integration  of  weaponeering, 
precision  point  mensuration,  and  collateral  damage  estimation, 
decreasing  the  speed  at  which  strike  planning  can  be  conducted. 

To  more  effectively  support  operational  mission  planning, 
particularly  at  U.S.  Strategic  Command,  the  J-ACE  v5.2.1  release 
also  provides  a  direct  interface  to  force-level  simulations.  The 
fidelity  is  adequate  for  studying  tactics,  training  evaluation, 
relative  missile  performance,  and  scenario  planning.  Additionally, 
U.S.  Strategic  Command  integrated  JAAM  into  the  Individual 
Combat  Aircrew  Display  System  and  the  Personal  Computer 
Debriefing  System  for  direct  use  for  tactics,  planning,  and  training 
at  operational  test  squadrons  for  fighters  and  bombers. 

INFORMATION  OPERATIONS  TOOLS  AND  CAPABILITIES 

To  address  an  emerging  Cyber  Operations  JMEM,  JTCG/ME 
re-deployed  the  following  Joint  Capability  Analysis  and 
Assessment  System  (JCAAS)  tools:  Computer  Network  Attack 
Risk  and  Effectiveness  Analyzer,  Network  Risk  Assessment  Tool, 
Communications  Radar  Electronic  Attack  Planning  Effectiveness 
Reference,  Effectiveness  of  Psychological  Influence  Calculator, 
and  Joint  Broadcast  Analysis  Tool.  JCAAS  will  provide  a  shared 
interface  for  operational  users  in  selecting  the  capabilities  to  best 
meet  given  objectives  based  on  effectiveness  derived  from  target 
vulnerability  and  capability  characteristics.  The  JCAAS  scope 
includes  weapon  characterization;  coordinating  test  and  target  data 
development;  testing  and  evaluation  of  cyber  data  standards;  and 
developing  a  new  database  schema  for  electronic  warfare  mission 
planning. 


JOINT  AIRCRAFT  SURVIVABILITY  PROGRAM  (JASP) 


The  Joint  Aircraft  Survivability  Program  (JASP)  office  funds 
development  of  techniques  and  technology  to  improve  the 
survivability  of  U.S.  military  aircraft.  Working  with  Joint  Staff, 
Service  organizations,  other  government  agencies,  and  industry, 
JASP  develops  new  capabilities  and  works  to  assure  they  are 
pursued  jointly  by  the  Services. 

DOT&E  sponsors  and  funds  JASP.  The  Naval  Air  Systems 
Command,  the  Army  Aviation  and  Missile  Command,  and  the 
Air  Force  Life  Cycle  Management  Center  charter  the  program. 
DOT&E  establishes  objectives  and  priorities  for  the  JASP  and 
exercises  oversight  of  the  program. 

JASP  is  supporting  the  Joint  Multi-Role  (JMR)  Technology 
Capabilities  Demonstration  (TCD)  program  as  a  member  of  the 


Platform  Integrated  Product  Team.  The  JMR  TCD  is  expected  to 
demonstrate  transformational  vertical  lift  capabilities  to  prepare 
the  DOD  for  developing  the  next-generation,  vertical  lift  fleet. 
JASP  was  a  driving  force  in  establishing  the  assumptions  and 
requirements  for  the  vulnerability  analysis  used  in  evaluating  the 
initial  three  government  model  prototypes. 

JASP  funded  66  multi-year  survivability  projects  and  delivered 
48  final  reports  in  FY14.  The  following  summaries  highlight 
selected  JASP  efforts  in  four  focus  areas:  susceptibility 
reduction,  vulnerability  reduction,  survivability  assessment,  and 
combat  damage  assessment. 
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SUSCEPTIBILITY  REDUCTION 

Multiple  Objective  Differential  Evolution  Smart-dispense 
Techniques  (MODEST).  The  Naval  Research  Laboratory 
(NRL)  in  conjunction  with  the  Naval  Surface  Warfare 
Center  -  Crane  Division  (NSWC-Crane)  and  the  Air  Force 
Research  Laboratory  are  developing  techniques  to  improve 
dispensing  infrared  countermeasures  (IRCM). 


Smart  Dispense  Concept 


MWS  -  Missile  Warning  System 


Countermeasure 
response 
tailored  to  this 
engagement 


They  have  leveraged  recent  advances  in  multiple  objective 
differential  evolution  to  develop  a  methodology/tool  to  obtain 
near-optimal,  smart-dispense  techniques  for  infrared  decoys. 

They  also  formulated  an  optimal,  non-linear  effectiveness 
measure  for  jam  codes  based  on  measurements  from  actual  test 
scenarios.  The  improved  dispense  algorithms  will  reduce  flare 
usage  and  improve  other  countermeasure  synergistic  effects.  The 
initial  plan  is  for  the  Navy  to  implement  new  dispense  techniques 
on  Navy/Marine  Corps  aircraft.  Other  services  plan  to  assess 
techniques  and  implement  as  needed. 

High  Resolution  Infrared  Clutter  Measurement  and  Analysis. 

The  Army  Communications-Electronics  Research,  Development 
and  Engineering  Center/Intelligence  and  Information  Warfare 
Directorate  is  studying  techniques  for  developing  a  cross-Service 
definition  of  clutter  and  creating  a  database  of  high-resolution 
narrow  field  of  view  (FOV)  clutter  for  all  Services.  The 
study  will  improve  the  government’s  ability  to  adequately 
define  and  test  the  effectiveness  of  missile  warning  systems 
to  pointer/ tracker  handoff.  The  project  intends  to  establish 
clutter-level  definitions  and  a  narrow  FOV/high-resolution  clutter 
database  to  support  development  of  tri-Service  directed  IRCM 
systems.  The  Army  Program  Manager  Aircraft  Survivability 
Equipment  is  providing  data  and  funding  to  develop  a  test 
tool  and  clutter  scene  injection  capability.  The  project  is  also 
leveraging  additional  data  collected  from  the  Advanced  Threat 
Infrared  Countermeasures  (AT1RCM)  program.  The  project 
will  result  in  more  reliable  and  effective  directed  IRCM  systems 
by  allowing  the  government  to  ensure  pointer  trackers  perform 
per  specification.  The  Army  will  use  the  capability  to  improve 
fielded  algorithms  for  ATIRCM  and/or  the  Common  Infrared 
Countermeasures  systems. 

Ultraviolet  Reflective  Coating.  The  NRL  is  leveraging 
commercial  technology  for  visibly-reflective  coatings  that  contain 
reflective  dielectric  multi-layer  flakes.  This  technology  will  help 
them  design  and  fabricate  a  coating  using  reflective  polymer 
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flakes  to  approach  100  percent  ultraviolet  reflectivity  without 
compromise  in  the  visible  and  infrared  wavelength  performance. 
NRL  will  test  and  evaluate  coating  performance  and  durability 
as  applied  to  representative  aircraft  surfaces.  Initial  methods 
to  disperse  polymer  flakes  into  coating  have  been  identified. 
NRL  has  completed  modeling  of  best-candidate,  commercially  - 
available  polymers  to  determine  preliminary  designs  for  multi¬ 
layer  polymer 
stacks.  NRL 
will  contract 
with  a 
commercial 
vendor  to 
fabricate 
multi-layer 
polymer 
stacks  using 
co-extrusion 
techniques 
based  on  NRL 
designs.  The 
improved 
coating 
will  make 
aircraft  less 
susceptible 
to  anti-aircraft 
systems. 

The  NRL 

and  Army  Aviation  Applied  Technology  Directorate  plan  to  test 
coatings  in  conjunction  with  other  IRCM  trials. 
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Airborne  Expendable  Countermeasure  (CM)  Velocity  Study. 

The  NSWC-Crane  started  this  project  in  FY14  to  determine  the 
critical  characteristics  in  airborne  expendable  CM  design  that 
effect  ejection 
velocity  in  order 
to  optimize 
performance. 

Demonstrating  a 
more  consistent, 
optimized 
velocity  will 
improve  decoy 
effectiveness 
and  aircraft 
survivability.  A 
consistent  ejection 
velocity  will 
give  accurate 
placement 
of  the  CM, 
greatly  assisting 
threat  defeat. 

Additionally,  once 
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Initial  gas  expansion  volume 
Cartridge  case  end  configuration 
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a  desired  ejection  velocity  is  obtained  and  parameters  that  affect 
it  are  identified,  a  common  impulse  cartridge  for  all  Services  will 
be  more  easily  achievable,  thereby  reducing  support  costs.  In  the 
next  phase,  NSWC-Crane  will  conduct  a  flight  test  to  determine 
how  tailored  flares  improve  effectiveness.  Methods  to  adjust 
ejection  velocity  will  be  used  to  enhance  effectiveness  of  current 
expendables. 

VULNERABILITY  REDUCTION 

JASP  vulnerability  reduction  projects  address  the  survivability 
of  the  crew  and  passengers,  as  well  as  the  aircraft  itself.  A 
portion  of  these  projects  have  focused  on  improving  armor 
and  developing  lighter-weight  opaque  and  transparent  ballistic 
protection  systems.  In  FY14,  projects  were  initiated  that  focus 
on  fuel  containment  technologies  and  their  related  fire  protection 
systems,  along  with  numerous  structures  and  materials  projects, 
including  self-healing  composites.  JASP  has  begun  to  explore 
occupant  seat  technology,  in  order  to  make  them  more  crash- 
worthy,  and  helicopter  transmission  technology,  to  make  them 
and  their  lubrication  systems  more  tolerant  to  ballistic  damage. 


depth.  Multi-hit  Transparent  Armor 

Additionally, 

the  principal  requirements  of  unhindered  visibility  for  situational 
awareness  and  defeat  of  a  designated  threat  with  a  multi-hit 
capability  have  to  be  considered. 

The  goal  of  this  project  was  to  reduce  total  system  weight,  while 
improving  performance  at  temperature  extremes  found  in  current 
combat  areas.  Variables  considered  for  improvement  included 
surface  treatments,  ply  layup  alterations,  and  edge  framing.  This 
was  a  successful  proof  of  concept  effort  achieving  systems  in 
the  five  pounds  per  square  foot  range  over  a  greater  temperature 


range,  compared  to  current  advanced  prototype  systems  weighing 
more  than  seven  pounds  per  square  foot. 

Unmanned  Aerial  Vehicle  (UAV)  Self-Sealing  Polymer.  As 

unmanned  aerial  vehicles  (UAVs)  become  more  prevalent  and 
greater  contributors  in  the  battlespace,  their  survivability  becomes 
more  crucial.  NAWC-WD  investigated  ways  to  improve 
the  survivability  of  the  MQ-8B  Fire  Scout  using  lightweight 
polymer  coatings  for  fuel  leak  mitigation.  UAVs  do  not  typically 
employ  any  type  of  fire  detection  and  suppression  systems,  nor 
system  hardening  to  protect  against  projectiles  or  other  threats. 
Applying  a  coating  that  self-seals  (closes  over  a  fragment  or 
projectile  penetration)  and  prevents  fuel  leakage  onto  a  hot 
surface  and  subsequent  possible  ignition  is  a  valuable  contributor 
to  vulnerability  reduction.  The  project  focused  on  applying  a 
coating  to  a  thin  aluminum  fuel  cell  tray  that  sits  above  the  engine 
bay  and  holds  the  aft  fuel  cell.  Test  scenarios  observed  different 
impact  angles,  coating  thickness,  and  configuration  (entry  or  exit 
side  of  fuel  tray).  Test  results  showed  that  coating  the  bottom 
of  the  fuel  tray  (entry  configuration)  with  a  nominal  coating  of  a 
polyurea  is  effective  in  preventing  hot  surface  fuel  ignition  at  a 
very  minimal  weight  (under  six  pounds)  and  material  cost  ($150). 
The  Fire  Scout  Program  Office  is  currently  evaluating  the  retrofit 
of  aircraft  during  depot-level  maintenance. 

Adaptive  Seat  Energy  Absorber  for  Enhanced  Crash  Safety. 

The  ARL- Vehicle  Technology  Directorate  is  investigating 
seat  energy  dispersion  technologies  for  aircraft  cockpits  to 
enhance  crash  safety.  The  focus  of  the  project  is  to  evaluate 
and  demonstrate  two  novel  energy  absorber  (EA)  technologies: 
Rotary  Magnetorheological  EA  with  Magnetic  Bias  (MREA) 
and  Magnetostrictive  Friction  EA  (MFEA).  In  the  MREA 
system,  a  fluid  that  changes  physical  properties  in  the 
presence  of  a  magnetic  field  is  used,  allowing  changes  in  the 
applied  current  to  alter  the  viscosity  of  the  fluid  for  optimum 
performance.  In  the  MFEA  system,  magnetostrictive 
materials  change  dimensions  when  magnetized  and  can  be 
used  to  adjust  the  normal  force  between  two  surfaces  to 
modulate  frictional  energy  absorption.  In  each  case,  the 
system  is  tuned  to  the  actual  occupant  weight  and  crash 
scenario  to  optimize  occupant  survival.  The  project  team 
has  made  substantial  progress  with  the  MREA  system — 
fabrication  of  a  seat-level  test  asset  is  near  completion.  The 
MFEA  component  device  is  being  redesigned.  Due  to  the 
limited  stroking  capability  of  magnetostrictive  technology, 
force  amplification  is  needed  to  meet  the  design  load 
requirements  for  the  crash  application. 

SURVIVABILITY  ASSESSMENT 

JASP  continues  to  develop  and  maintain  survivability  assessment 
tools  that  are  widely  used  by  acquisition  programs  for  analyses 
of  alternatives,  design  studies,  and  specification  development  and 
compliance.  JASP  also  supports  the  test  community  to  assess 
survivability  in  pilot  training  exercises  and  countermeasure 
effectiveness. 

Improved  Fire  Modeling.  Based  on  testing  and  analysis, 
fire  has  been  identified  as  the  largest  contributor  to  aircraft 


Improved  Advanced  Survivable  Canopy  Transparent 
Armor.  The  Army  Research  Laboratory  (ARL)-Weapons  and 
Materials  Research  Directorate  teamed  with  Naval  Air  Warfare 
Center  -  Weapons  Division  (NAWC-WD)  to  work  this  project, 
which  completed  in  FY14.  Transparent  armor  is  typically  made 
of  several  layers  of  materials  separated  by  polymer  interlayers. 
The  outer  layer  is  usually  a  type  of  glass,  such  as  soda-lime 
silica  or  borosilicate,  and  the  inner  plies  are  usually  a  polymer, 
such  as  a 
polycarbonate 
or  polyurethane 
with  a  thicker 
interlayer. 

The  use  of 
these  different 
materials 
induces 
inherent 
stresses  due 
to  the  thermal 
expansion 
mismatches 
across  the 
material’s 
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vulnerability.  For  many  years,  JASP  has  supported 
development  of  the  Fire  Prediction  Model  (FPM)  to  provide 
a  tool  for  analysts  and  testers  to  understand  and  mitigate  this 
vulnerability.  JASP  currently  has  three  projects  addressing  fire 
modeling  deficiencies.  The  Air  Force  96th  Test  Group  (96TG) 
is  developing  a  hydrodynamic  ram  engineering  model  that  will 
provide  accurate  fuel  spurt  characteristics  to  determine  the 
likelihood  of  fire  ignition.  The  ARL/Survivability  Lethality 
Analysis  Directorate  (ARL/SLAD),  with  participation  from 
the  96TG  and  NAWC-WD,  is  conducting  a  review  of  all  of  the 
documents  and  test  data  referenced  in  the  FPM  Analyst  and 
User  Manuals  to  identify  discrepancies  between  the  documents 
and  the  FPM  code.  In  the  third  project,  Next  Generation  Fire 
Modeling  Plan,  the  96TG,  in  partnership  with  NAWC-WD  and 
the  Air  Force  Life  Cycle  Maintenance  Center,  is  generating 
a  long-term  plan  to  guide  the  development  of  a  credible  and 
validated  dry  bay  fire  assessment  capability  for  armor-piercing 
incendiary  and  fragment  threats.  These  projects  all  affect  the 
development  of  the  next  generation  FPM  that  will  be  used  for 
live  fire  test  planning  and  survivability  assessments  of  future 
combat  vehicles,  providing  a  better  understanding  of  the 
vulnerability  of  these  systems  to  combat  initiated  fires. 

Improved  Target  Geometries.  Target  descriptions 
(geometries)  for  aircraft  vulnerability  analyses  have  grown  so 
detailed,  with  attendant  enormous  computer  run  times,  that 
they  restrict  the  ability  for  development  programs  to  conduct 
analyses  in  time  to  affect  design  decisions.  Interrogation,  via 
ray  tracing  methods,  of  target  geometric  models  is  a  cornerstone 
of  vulnerability  assessments.  JASP  has  funded  two  projects 
to  improve  ray 
tracing  and 
target  geometry 
optimization. 

Ray  Tracing 
Speed 

Enhancements 
is  investigating 
several  ways 
to  speed  up 
the  ray  tracing 
process.  The  Air 
Force  Life  Cycle 
Maintenance  Center  is  leading  this  project  with  participation 
from  ARL/SLAD.  Benchmark  testing  accomplished  to  date  has 
demonstrated  a  30  percent  decrease  in  run  times.  The  enhanced 
ray  tracing  algorithms  will  be  integrated  into  the  Computation 
of  Vulnerable  Area  Tool  (COVART)  for  use  on  future 
vulnerability  assessments.  The  Target  Geometry  Optimization 
Utility  project  is  investigating  ways  to  optimize  the  target 
geometry.  The  goal  is  to  reduce  files  sizes  by  33  percent  while 
changing  presented  area  estimates  by  less  than  1  percent. 
NAWC-WD  is  leading  this  project  with  participation  from 
ARL/SLAD.  The  results  of  this  project  will  be  distributed  with 
COVART  as  part  of  the  Defense  Systems  Information  Analysis 
Center  Vulnerability  Toolkit  for  use  on  future  vulnerability 
assessments. 


Imaging  Infrared  (HR)  Threats.  JASP  has  three  related 
projects  to  address  countermeasures  for  the  challenging  Imaging 
Infrared  (IIR)  threat.  In  the  Physics-Based  1RCM  Modules 
project,  NSWC-Crane  investigated  the  specific  data  and  level 
of  detail  required  to  model  flares  in  an  IIR  environment.  This 
project  developed  formats  for  physics-based  models  for  both 
pyrotechnic  and  pyrophoric  lRCMs  that  are  integrated  in  the 
Fast  Line-of-Sight  Imagery  for  Targets  and  Exhaust  Signatures 
(FLITES)  software.  FLITES  is  an  industry  standard  scene 
generation  software  package.  This  new  format  is  referred  to  as 
the  Dynamic 
Particle  Flare 
representation. 

In  the  Flare 
Aerodynamic 
Modeling 
Environment 
and  Tri-Service 
Flare  Database 
Modernization 
project, 

NSWC-Crane 
is  updating 
the  Flare 
Aerodynamic 
Modeling 
Environment 
model  to  add 

3-D  aircraft  flow  fields  and  modernize  the  Tri-Service  Flare 
Database  so  it  can  accommodate  flare  models  in 
the  Dynamic  Particle  Flare  format.  In  the  Imaging 
Flare  Models  for  Missile  Simulations  project, 
the  NRL,  with  NSWC-Crane  participation,  will 
produce  12  validated  flare  models  in  the  Dynamic 
Particle  Flare  format  that  are  compatible  with  the 
FLITES  scene  generation  software.  This  project 
will  also  develop  the  Flare  Imagery  Analysis  Tool 
to  generate  radiometrically-  and  spatially-correct 
imagery  of  flares  used  in  FLITES.  These  projects 
will  provide  tools,  available  through  the  Defense 
Systems  Information  Analysis  Center,  to  evaluate  the 
effectiveness  of  advanced  flares  against  IIR  threats. 

COMBAT  DAMAGE  ASSESSMENT 

JASP  continued  to  support  the  Joint  Combat  Assessment  Team 
(JCAT)  in  FY14.  JCAT  is  a  team  of  Air  Force,  Army,  and  Navy 
personnel  deployed  in  support  of  combat  operations.  JCAT 
continued  its  operation  in  Afghanistan  with  full-time  deployments 
in  Regional  Commands  -  South,  Southwest,  and  East.  Iraq  and 
other  areas  of  the  world  were  supported  remotely  or  by  rapid 
deployment  from  Afghanistan  or  the  Continental  U.S. 

JCAT  inspects  damaged  and  destroyed  aircraft,  acquires 
maintenance  records,  and  conducts  interviews  with  aircrew  and 
intelligence  personnel  to  develop  an  accurate  and  comprehensive 
assessment  of  each  aircraft  combat  damage  event.  They  provide 
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consultation  to  weapons,  tactics,  and  logistics  personnel  and 
comprehensive  briefings  to  commanders  in  charge  of  daily 
air  operations.  These  efforts  inform  battlefield  commanders, 
allowing  them  to  adjust  operational  tactics,  techniques,  and 
procedures  based  on  accurate  threat  assessments.  As  of 
August  31,  2014,  the  JCAT  had  initiated  131  and  completed  69 
aircraft  combat  damage  assessments  in  FY14. 

The  JCAT  strengthened  aircraft  combat  damage  incident 
reporting  in  the  Services  and  the  DOD.  The  Combat  Damage 
Incident  Reporting  System  is  the  repository  for  all  U.S. 
aircraft  combat  damage  reports.  JCAT  worked  with  OSD  and 
USCENTCOM  to  continue  collecting  combat  incident  reports 
from  USCENTCOM’s  databases  to  more  quickly  identify, 
assess,  document,  and  distribute  aircraft  combat  damage 
incident  data  to  the  Services  and  DOD.  JASP  and  the  Office 


of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems 
Engineering  submitted  a  revision  to  DOD  Instruction  5000.02, 
released  in  November  2013,  which  included  combat  data 
reporting  requirements  for  major  weapon  systems.  Language 
providing  additional  guidance  for  the  aircraft  combat  damage 
reporting  process  is  included  in  the  draft  revision  of  the  Defense 
Acquisition  Guide. 

The  JCAT  trains  the  U.S.  aviation  community  on  potential 
aircraft  threats  and  combat  damage.  This  training  includes,  but 
is  not  limited  to:  capabilities  briefs;  intelligence  updates;  recent 
“shoot-down”  briefs  to  discuss  enemy  tactics,  techniques,  and 
procedures;  and  the  combat  damage  collection  and  reporting 
mentioned  above.  The  attendees  include  aircrews,  maintenance 
personnel,  intelligence  sections,  Service  leadership,  industry,  and 
coalition  partners. 


JOINT  LIVE  FIRE  (JLF) 


The  purpose  of  the  Joint  Live  Fire  (JLF)  program  is  to  test 
fielded  systems,  identify  vulnerable  areas,  and  understand 
damage  mechanisms  to  provide  the  information  needed  for 
design  changes,  modifying  tactics,  techniques,  and  procedures, 
or  improving  analytical  tools.  The  need  for  these  tests  results 
from  systems  being  exposed  to  new  threats;  being  used  in 
new,  unanticipated  ways;  or  being  operated  in  new  combat 
environments,  thereby  requiring  an  updated  assessment  of  their 
performance. 

JLF  supplements  LFT&E  of  systems  by  testing  systems  against 
new  threats  the  requirements  community  did  not  anticipate  during 
original  development  or  against  old  threats  employed  in  new 
ways.  The  rocket-propelled  grenade  (RPG)  is  an  example  of  a 
threat  employed  differently  than  initially  intended.  Originally 
developed  as  an  anti-tank  or  anti-personnel  weapon,  hostile 
forces  in  combat  often  use  the  RPG  as  an  anti-aircraft  weapon. 

AIRCRAFT  SYSTEMS  PROGRAM 

JLF-Air  completed  six  test  series  in  FY14.  Seven  of  this  year’s 
projects  will  continue  into  FY15.  Below  are  selected  examples 
of  this  year’s  projects. 

Advanced  Hit  Efficiency  and  Destruction  (AHEAD) 
Sub-projectile  Characterization  Testing.  This  project 
determined  the  penetration  characteristics  for  a  modern  threat, 
anti-aircraft  artillery  projectile — the  air  burst  Advanced  Hit 
Efficiency  and  Destruction  (AHEAD)  projectile.  The  penetration 
characteristics  of  its  tungsten  sub  projectiles  were  measured 
for  three  thicknesses  and  three  obliquities  of  aluminum  panels 
representative  of  aircraft  structures  and  flight  critical  components. 
Data  will  help  the  development  of  a  penetration  model  to 
effectively  model  air  burst  munitions  sub-projectiles,  providing 
an  analysis  capability  presently  unavailable  in  commonly  used 
aircraft  vulnerability  codes,  such  as  COVART  and  the  Advanced 
Joint  Effectiveness  Model  (AJEM). 


ARL/SLAD  conducted  248  tests  in  FY14  and  calculated  the 
V50  (the  velocity  at  which  50  percent  of  the  projectiles  penetrate 
the  target  material)  ballistic  limits,  and  associated  80  percent 
confidence  intervals  for  each  material  thickness-obliquity  pairing. 
Additional  data  collected  included  impact  orientation,  residual 
mass,  and  flash  characterization. 


AHEAD  sub-projectile  and  its  damage 


Man-portable  Air  Defense  System  (MANPADS)  Residual 

Fuel.  This  project  collected  blast  pressures  produced  by 
stationary  and  moving  Man-portable  Air  Defense  System 
(MANPADS)  missiles  when  a  portion  of  the  missile’s  rocket 
motor  fuel  remains  at  the  time  of  warhead  detonation.  Results 
will  be  coupled  with  blast  pressures  measured  during  previous 
JLF-Air  tests  and  used  to  further  improve  MANPADS  threat 
models  used  in  aircraft  vulnerability  assessment  codes,  such 
as  COVART  and  AJEM,  as  well  as  damage  prediction  and 
assessment  tools  such  as  LS-DYNA  and  the  Combat  Assessment 
Tool. 
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The  NAWC-WD,  Weapons  Survivability  Laboratory,  located  at 
China  Lake,  California,  conducted  testing.  They  collected  42 
pressure  measurements  at  radial  distances  of  12,  20,  30,  40,  and 
72  inches  from  the  MANPADS  warhead  centroid  for  2  stationary 
detonation  events  and  a  single  detonation  event  with  a  surrogate 
MANPADS  missile  traveling  at  1,600  feet  per  second. 

Crew  Compartment  Fire  Survivability.  This  project  collected 
data  to  determine  the  effects  of  fuel  fires  on  temperatures, 
oxygen  depletion,  carbon  monoxide,  and  other  toxic  gases.  The 
aircraft  survivability  community  can  use  these  data  to  support 
assessments  of  physiological  hazards  within  military  aircraft  crew 
compartments  as  a  function  of  time  and  distance  from  the  fire 
source  under  realistic  ventilation  conditions.  Collected  data  will 
establish  baseline  assumptions  for  future  crew  casualty  models 
and  analysis  efforts. 


Crew  Compartment  Fire  Testing 


ARL/SLAD,  located  at  Aberdeen,  Maryland,  conducted  testing. 
They  completed  48  tests  using  Design  of  Experiments  (DOE) 
techniques  along  with  several  excursion  tests  in  FY14.  The 
DOE  test  matrix  was  based  on  factors  of  fluid  pressure,  piping 
diameter,  ventilation  configuration,  mean  airflow,  and  fire 
location.  Excursion  tests  used  baseline  (Halon  1301,  C02) 
hand-held  fire  extinguishers  and  potential  replacement  fire 
extinguishers  with  selected  repeats  using  hydraulic  fluid  instead 
of  aircraft  fuel. 

GROUND  SYSTEMS  PROGRAMS 

Sustained  Fire-Start  from  Near-Field  Detonations  of  Blast/ 
Frag  Weapons.  The  JLF-Ground  program  funded  the  Air  Force 
Research  Laboratory  and  ARL  to  conduct  a  study  of  sustained 
fire  start  research  as  it  relates  to  weapons  effects  against  ground 
vehicle  targets.  Initiation  of  sustained  fires  is  one  of  several 
lethal  mechanisms,  which  can  impart  significant  levels  of  damage 
to  a  materiel  target.  Experiments  are  being  conducted  on  fuel 
tanks  at  varying  ranges  from  a  threat,  using  optical  pyrometry  to 
characterize  the  fireball.  Physical  understanding  and  empirical 
data  will  be  transitioned  to  fire  start  prediction  models  improving 


developer  and  evaluator  predictions  of  effectiveness,  especially 
catastrophic  kill  of  ground  targets. 

Adobe  Wall  Combatant  Commander  Validation  Test.  The 

Army  conducted  a  series  of  tests  to  evaluate  the  protection 
provided  by  an  adobe  wall  from  fragments  from  MK82  and 
HELLFIRE  Romeo  warheads.  These  tests  will  provide 
benchmark  data  for  warhead  fragments  from  two  weapons  that 
undergo  extensive  operational  use  against  high-interest  targets 
in  theater.  The  results  of  this  testing  will  provide  understanding 
of  the  methodology  not  previously  available,  along  with  updated 
lethality  data  of  warheads  that  will  improve  effectiveness  and 
collateral  damage  analyses  for  Combatant  Commanders. 

Assessment  of  Peepsite  Generation  2  (PG2)  Fleadform 
for  Ballistic  Testing  of  Helmets.  This  project  assessed  the 
capability  of  the  PG2  headform  to  measure  back  face  deformation 
(BFD)  for  helmet  testing  compared  to  the  National  Institute 
of  Justice  (NIJ)  headform.  The  current,  single-sized  NIJ 
headform  has  limitations  due  to  its  current  design,  which  utilizes 
two  clay  channels  and  four  aluminum  pillars  that  interfere 
with  BFD.  Phase  I  of  testing  encompassed  48  subtests  (each 
subtest  had  3  shots  per  helmet)  comparing  BFD  repeatability. 
Phase  II  assessed  test-induced  variation  and  explored  additional 
capabilities  of  the  PG2  headform.  Phase  II  was  completed  with 
48  subtests  (each  subtest  had  one  shot  per  helmet). 


National  Institute  of  Justice  Peepsite  Generation  2  (PG2) 
(NIJ)  Back  Face  Deformation  Back  Face  Deformation 


(BFD)  Headform  (BFD)  Headform 

Enhanced  Modeling  of  Behind-Armor  Debris  from  Kinetic 
Energy  Penetrators.  ARL  conducted  tests  of  medium-caliber 
kinetic  energy  penetrators  to  support  the  modeling  of  the 
behind-armor  debris  (BAD)  from  the  penetrators.  Physical 
characteristics  (mass,  velocity,  spatial  distribution,  shape  factor) 
of  the  residual  penetrating  fragments  were  collected  with  flash 
x-ray,  plywood/Celotex  bundles,  and  witness  panels.  ARL  will 
use  these  data  to  enhance  the  current  BAD  algorithm  in  the 
Army’s  primary  vulnerability/lethality  model  by  producing  a 
more  accurate  physical  representation  of  medium-caliber  residual 
fragments. 
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Testing  and  Evaluation  of  Current  U.S.  Army  Body  Armor 
against  Emerging  Threats.  This  project  quantifies  and 
compares  the  penetrative  capability  of  selected  emerging  or 
persistent  non-standard  threats  against  particular  U.S.  Army 
body  armor  protection  levels.  Data  collected  of  resistance 
to  penetration  and  ballistic  protection  limits  will  determine  a 
logistic  regression  prediction  of  the  V05,  the  V50,  and  the  V95 
(velocities  at  which  5  percent,  50  percent,  and  95  percent  of 
projectiles  penetrate  the  armor,  respectively)  ballistic  limits. 

Data  from  the  ballistic  protection  limit  (V50)  will  include  BAD 
characterization  against  a  ballistic  gelatin  backing.  The  selected 
threats  for  ballistic  testing  are  undergoing  cross-sectional 
profile  characterization,  including  optical  microscopy,  digital 
photography,  evaluation  of  material  composition  (elemental  and 
alloy  analysis),  cross-section  dimensional  characterization,  and 
hardness  testing.  ARL/SLAD  and  the  Aberdeen  Test  Center  are 
currently  performing  the  resistance  to  penetration  and  ballistic 
limit  testing  for  this  project. 

Environmental  Aging  Effects  on  the  Protection  Levels  of 
Armor.  The  USMC  Combat  Development  Command  (MCCDC) 
conducted  a  series  of  ballistic  and  material  tests  on  aluminum 
specimens  removed  from  original  manufactured  Amphibious 
Assault  Vehicle  hulls  to  understand  potential  effects  of  age  that 
can  affect  a  variety  of  material  properties.  With  an  anticipated 
platform  upgrade 
utilizing  existing 
hulls,  knowledge 
of  present 
protection  levels 
is  imperative. 

Plates  were 
removed  from 
several  locations 
that  had  been 
exposed  to 
varying  heat 
levels  over 
their  lifetime. 

MCCDC 
executed  V50 

tests  for  several  penetrators  at  various  obliquities  to  determine 
if  the  plates  still  met  the  original  minimum  requirements  for 
armor  protection.  MCCDC  also  performed  material  tests  (tensile, 
hardness,  metallographic,  inter-granidar  corrosion  susceptibility, 
and  Charpy  impact)  to  assess  current  material  properties  and 
extent  of  change  due  to  aging  that  occurred.  Testing  showed  that 
while  the  materials  incurred  significant  aging,  the  plates  still  meet 
the  original  minimum  protection  requirements  of  the  armor. 

Collaborative  Validation  of  Mandible  Blunt  Impact 
Methodology.  To  better  evaluate  the  efficacy  of  mandible 
protection  systems,  this  project  obtained  operationally-realistic 
blunt  impact  loadings  that  vehicle  occupants  may  experience. 
Data  collected  from  instrumented  headforms  included,  but  were 
not  limited  to:  mandible  and  nasal  force,  head  acceleration,  and 


Test  Specimen  Cut  from  Amphibious 
Assault  Vehicle  (AAV)  Hull 


neck-bending  moments.  ARL  used  these  values  to  determine 
whether  existing  simplified  test  apparatuses,  such  as  a  drop  tower 
or  pendulum,  could  be  used  to  evaluate  blunt  impact  loadings. 


Helmet  with  Mandible  Protection  under  Test 


Characterizing  the  Penetration  of  an  Explosively-Formed 
Penetrator  Mine.  The  Army  conducted  testing  to  determine 
the  lethality  of  an  explosively-formed  penetrator  mine  and  its 
lethality  against  underbody  armor.  The  data  obtained  from 
these  tests  provide  the  ability  to  better  assess  the  vulnerabilities 
of  armored  combat  vehicles  to  the  penetration  of  this  mine. 
Penetration  tests  documented  the  penetrator  formation,  flight 
characteristics,  and  penetration  performance  into  semi-infinite 
and  spaced  rolled  homogenous  steel  targets.  Additionally,  BAD 
tests  determined  the  amount  of  fragmentation  resulting  from  both 
the  residual  threat  and  the  spall  production. 

Irregular  Fragment  Penetration  Characterization  and 
Model  Validation.  The  objective  of  this  project  is  to  identify 
and  validate  the  most  appropriate  Fast  Air  Target  Encounter 
Penetration  model  representation  of  irregular  fragments.  To 
accomplish  this,  ARL  collected  and  laser  scanned  approximately 
125  irregular  fragments  from  an  OF540  152  mm  artillery  shell  to 
develop  multiple  shape  characterizations. 


Irregular  Fragment 


Scanned  Fragment 
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ARL  will  fire  the  125  fragments  against  steel  plates  at  a 
sufficient  distance  to  allow  random  tumbling  to  occur.  Fragment 
orientation,  striking  and  residual  mass,  and  velocity  will  be 
collected.  Pre-shot  penetration  characterizations  of  the  multiple 
shape  models  will  then  be  compared  to  the  test  data  and  a 
determination  made  on  the  most  appropriate  shape  factor  model. 
This  project  will  improve  future  LFT&E  through  improved 
pre-test  prediction  capabilities  and/or  reduced  number  of  tests 
required  due  to  increased  confidence  in  modeling. 

SEA  SYSTEMS  PROGRAM 

The  JLF  Sea  Systems  Program  made  significant  progress  in  FY14 
towards  improving  the  capability  to  assess  the  survivability  of 
submarines  and  surface  ships.  These  projects  benefit  ship  and 
submarine  acquisition  programs,  as  well  as  the  fleet  of  fielded 
U.S.  Navy  vessels. 

Large  Volume  Shipboard  Space  Fire  Protection.  Large 
volume  spaces  such  as  aircraft  hangars  on  aircraft  carriers  and 
vehicle  stowage  areas  on  amphibious  ships  are  frequently  used  to 
stow  large  quantities  of  ordinary  combustible  (Class  A)  material. 
These  spaces  are  usually  protected  with  overhead  Aqueous  Film 
Forming  Foam  (AFFF)  sprinkling  systems,  which  testing  has 
shown  have  limited  effectiveness  against  shielded,  large  Class  A, 
vehicle,  and  weapon-induced  fires.  As  part  of  this  project,  a  fire 
hazard  analysis  determined  that  supplementing  legacy  AFFF 
sprinkling  systems  with  bulkhead-mounted  AFFF  monitors  or 
water/ foam  cannons  could  potentially  improve  the  system’s 
capability  against  these  fires.  During  FY14,  NRL  designed  and 
installed  a  water/foam  cannon  system  and  monitor  control  system 
in  the  hangar  bay  test  area  of  the  ex-USS  Shadwell  Full-Scale 
Fire  Research  and  Test  Ship.  The  system  was  commissioned  and 


preliminary 
fire  testing 
completed  in 
October  2014. 

In  FY15,  NRL 
will  conduct 
full-scale 
testing  against 
large  and 
obstructed 
Class  A  and 
pool  and 

running-liquid  fuel  fires  to  demonstrate  the  capability  of  the 
system  against  these  difficult  fires.  Testing  will  include  low 
visibility  conditions. 

Deep-Depth  Underwater  Explosion  Testing  of  Asymmetric 
Cylinders.  Future  submarine  pressure  hull  design  will 
likely  include  structural  features  that  will  influence  the 
primary  mode  of  collapse  when  subjected  to  a  deep-depth 
underwater  explosion.  This  project  will  provide  a  new  and 
experimentally-based  understanding  of  asymmetric  pressure 
hull  failure  modes  and  an  assessment  of  the  effects  of  unique 
geometries  on  lethal  depth.  This  new  understanding  will  improve 
future  submarine  vulnerability  assessments.  Additionally, 
the  data  set  provided  by  this  program  will  be  used  to  validate 
computational  models,  leading  to  increased  confidence  in  lethal 
depth  predictions  for  asymmetric  pressure  hulls.  During  FY14, 
the  NSWC  -  Carderock  Division  designed  test  cylinders  and 
identified  shot  geometries.  They  will  fabricate  the  test  cylinders 
in  FY15  and  test  them  during  FY16. 


LFT&E  SPECIAL  INTEREST  PROGRAMS 


PERSONNEL  PROTECTION  EQUIPMENT 

DOT&E  continues  to  exercise  oversight  over  personal 
protective  equipment.  The  U.S.  Special  Operations  Command 
(USSOCOM)  and  Services  continue  to  implement  rigorous, 
statistically-principled  testing  protocols  approved  by  DOT&E 
for  hard  body  armor  inserts  and  military  combat  helmets.  In 
partnership  with  USSOCOM  and  the  Services,  DOT&E  has 
begun  developing  a  protocol  for  soft  armor  vest  testing  that  will 
standardize  testing  of  soft  armor  vests  and  require  them  to  meet 
rigorous  statistical  measures  of  performance.  This  represents 
the  final  commodity  area  (in  addition  to  hard  armor  plates 
and  combat  helmets)  for  which  DOT&E  intends  to  develop  a 
statistically-based  protocol  for  future  testing. 

The  National  Research  Council  (NRC)  completed  its  independent 
review  of  helmet  testing  protocols.  The  NRC’s  final  report 
contained  several  recommendations  that  closely  align  with 
ongoing  efforts  to  improve  helmet  performance  and  testing  within 
the  Department.  DOT&E  has  also  modified  the  relevant  protocols 
to  reflect  recommendations  from  the  NRC  and  other  external 
reviews.  This  revised  protocol  reduces  government  risk  and 
achieves  simplification,  both  of  which  the  NRC  recommended. 


As  noted  by  the  NRC  in  their  final  report  on  helmet  testing, 
a  clear  scientific  link  to  the  modes  of  human  injury  from 
ballistic  impact,  blast,  and  blunt  trauma  do  not  exist.  This  is 
a  serious  limitation  for  the  test  and  evaluation  of  all  personal 
protective  equipment.  DOT&E  is  monitoring  a  JLF-funded 
effort  to  establish  injury  risk  criteria  for  one  type  of  injury  due 
to  behind-helmet  blunt  trauma;  the  initial  results  of  this  study 
should  be  available  next  year.  DOT&E  is  also  monitoring  a 
multi-year  Army  program  to  investigate  behind-helmet  blunt 
trauma,  determine  injury  mechanisms  and  risks,  and  develop 
an  injury  criterion  that  can  be  used  for  helmet  testing.  DOT&E 
is  overseeing  and  participating  in  the  Army’s  effort  to  improve 
helmet  test  mount  headforms  by  developing  multiple-sized 
headforms  to  replace  the  single-sized  headform  currently  used 
to  test  all  helmet  sizes.  This  year,  the  Army  completed  initial 
testing  of  a  multiple-sized  headform  and  is  determining  how 
to  implement  it  in  future  testing.  DOT&E  will  work  with 
USSOCOM  and  the  Services  to  update  personal  protective 
equipment  test  standards  and  procedures  to  incorporate  the 
results  of  these  efforts. 
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WARRIOR  INJURY  ASSESSMENT  MANIKIN  (WIAMAN) 

DOT&E  continued  its  oversight  of  the  Warrior  Injury 
Assessment  Manikin  (WIAMan)  project,  an  Army-led  research 
and  development  effort  to  design  a  biofidelic  prototype 
anthropomorphic  test  device  (ATD)  specifically  for  underbody 
blast  testing.  In  FY14,  the  project  faced  challenges  as  a  result 
of  a  disruption  in  the  supply  of  post-mortem  human  subjects 
(PMHS)  available  for  medical  research.  This  disruption  was  the 
consequence  of  a  combined  state  and  federal  investigation  into 
the  practices  of  a  PMHS  supplier  who  provided  specimens  to 
both  DOD  and  non-DOD  research  facilities  all  over  the  country, 
and  resulted  in  the  temporary  suspension  of  PMHS  testing  in  the 
project.  The  WIAMan  Project  Management  Office,  along  with 
the  ARL  Health  and  Safety  Office  with  guidance  from  the  Army 
Medical  Research  and  Material  Command,  implemented  an 
improved  protocol  for  vetting  specimen  suppliers  to  minimize  the 
probability  of  similar  disruptions  in  the  future.  In  FY14,  DOT&E 
contributed  financially  to  this  process,  which  will  ultimately 
establish  a  DOD-trusted  PMHS  supply  network  for  future  PMHS 
testing  conducted  within  the  Department. 

The  restructuring  of  the  WIAMan  medical  research  program 
DOT&E  reported  in  FY13  resulted  in  significant  progress  on 
this  front  in  FY14.  In  addition  to  developing  initial  biofidelity 
response  corridors  to  guide  the  design  of  the  ATD,  the  project 
completed  concepts  for  the  ATD  and  its  instrumentation/ 
data  system.  Another  key  accomplishment  in  FY 14  was  the 
creation  of  an  injury  research  plan  responsive  to  LFT&E  needs 
as  expressed  in  the  2010  DOT&E  issue  paper  on  this  subject; 
the  original  medical  research  approach  had  not  previously 
been  refined  in  a  manner  suitable  for  the  development  of  an 
LFT&E-specific  research  plan.  This  plan  also  incorporates  data 
derived  from  the  WIAMan  project’s  efforts  to  examine  imagery 
from  combat  injuries  to  better  define  the  specific  types  of  injuries 
that  should  be  assessed,  and  ultimately  prevented,  when  using 
the  new  ATD  in  underbody  blast  LFT&E.  An  example  of  the 
success  of  the  current  approach  is  the  emergence  of  new  injury 
probability  curves  for  the  foot  and  ankle  using  velocity  data 
derived  from  the  LFT&E  environment,  in  conjunction  with 
examination  of  x-rays  of  lower  limb  injuries  that  occurred  during 
underbody  IED  events  in  combat.  The  combat  data  trends 
revealed  that  foot  and  ankle  fractures  are  prevalent  injuries  in 
theater.  In  addition,  the  radiographic  images  gave  the  medical 
researchers  insights  into  what  kinds  of  fractures  are  representative 
of  combat  injuries  when  conducting  their  experiments,  using 
inputs  representative  of  the  loading  imparted  through  the  vehicle 
and  into  the  occupant  during  a  live  fire  test.  The  timely  and 
ongoing  provision  of  de-identified  medical  imagery  and  data 
to  the  project  are  critical  to  complementing  the  refined  injury 
research  plan  to  ensure  the  injuries  produced  by  the  medical 
researchers  are  representative  of  injuries  occurring  in  combat 
operations.  Such  knowledge  will  continue  to  facilitate  the 
success  of  the  WIAMan  project  as  it  significantly  improves  the 
Department’s  underbody  blast  LFT&E  capabilities  and  builds 
better,  more  protective  vehicle  platforms  for  our  Soldiers,  Sailors, 
Airmen,  and  Marines. 


SMALL  BOAT  SHOOTERS'WORKING  GROUP 

Small  boats  represent  a  growing  threat  class  to  ships  operating 
in  littoral  waters.  They  have  been  identified  as  a  required 
class  of  targets  for  a  wide  variety  of  tactical  missile,  rocket, 
and  gun  weapon  programs  on  DOT&E  oversight,  including 
25  mm,  30  mm,  and  57  mm  ammunition;  HELLFIRE, 

Joint  Air-to-Ground  Missile  (JAGM),  Evolved  SeaSparrow 
Missile,  Rolling  Airframe  Missile,  SDB  II.  They  are  also  threats 
of  concern  for  ships,  including  the  LCS  and  the  DDG  1000. 

On  September  11,  2014,  DOT&E  sponsored  the  third  meeting  of 
the  Small  Boat  Shooters’  Working  Group  at  the  Naval  Surface 
Warfare  Center  in  Dahlgren,  Virginia,  which  55  warfighters, 
evaluators,  and  weapons  designers  from  all  Services  attended. 

The  objective  of  the  meeting  was  to:  1)  examine  the  general 
nature  of  the  small  boat  threat  in  littoral  waters;  2)  summarize 
the  threat  classes  and  available  targets  and  models  available  for 
ammunition,  rocket,  and  tactical  missile  weapon  systems;  and 
3)  attempt  to  “harmonize”  various  LFT&E  and  other  operational 
test  approaches  among  the  various  programs/Services  by  sharing 
the  breadth  of  test  and  evaluation  options  available  to  evaluators. 

An  important  success  story  reported  at  the  Small  Boat  Shooters’ 
Working  Group  is  the  sudden,  widespread  use  of  a  standard  Fast 
Attack  Craft  (FAC)  boat  target,  the  decommissioned  Coast  Guard 
41-foot  patrol  boat  (CG-41).  Two  years  ago,  DOT&E  learned  the 
Coast  Guard  was  phasing  out  this  boat  class,  and  determined  that 
it  would  likely  provide  a  good  interim  structural  and  mechanical 
live  fire  surrogate  for  FAC  threats.  DOT&E  rapidly  informed 
the  Services,  and  worked  with  them  to  alter  live  fire  plans  to 
accommodate  the  test  articles.  The  Services  responded  and 
reported  at  the  meeting  that  they  obtained  26  of  the  54  available 
ex-CG-4 1  boats  for  tests  of  HELLFIRE,  JAGM,  the  SDB  II,  F-35 
Joint  Strike  Fighter  ammunition,  and  LCS.  Most  of  these  boats 
were  obtained  free  of  charge  from  the  Coast  Guard. 


U.S.  Coast  Guard  CG-41 


Briefings  this  year  centered  on  the  nature  of  the  small  boat  threat; 
the  availability  of  targets  and  lethality  models  representing 
those  threats;  and  the  data  collection,  test  techniques,  and 
instrumentation  that  have  been  applied  to  small  boats.  Attendees 
also  discussed  upcoming  test  plans  for  the  HELLFIRE  R-9E 
missile,  JAGM,  Griffin  missile,  SDB  II,  and  F-35  Joint  Strike 
Fighter  ammunition  against  a  variety  of  FAC  and  Fast  Inshore 
Attack  Craft  threats.  Of  special  interest  were  test  results  from 
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HELLFIRE  Romeo  missiles  fired  against  the  ex-Coast  Guard 
CG-41  FAC  surrogate,  and  HELLFIRE  Longbow  missiles 
vertically  fired  from  a  ship  against  ex-Coast  Guard,  25-foot 
“Guardian”  boats. 

Two  of  the  central  observations  from  the  group  are  the  need  for 
a  broader  variety  of  surrogate  small  boat  targets  and  better  live 
fire  data  collection  from  operational  test  events.  Participants 


stressed  the  need  for  evaluating  the  likelihood  of  mobility  kills 
in  conjunction  with  live  fire  evaluations  against  small  boats, 
since  this  is  the  most  clearly  observable  kill  mode  in  fleet 
exercises.  Evaluators  also  encouraged  the  increased  use  of  small, 
easy-to-place  “in-situ”  camera  and  overpressure  measurement 
packages  on  the  boats  in  order  to  better  observe  and  record  the 
types  of  damage  to  target  boats  during  weapons  tests. 
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Cybersecurity 


SUMMARY 


DOT&E  cybersecurity  efforts  in  FY14  included  16  Combatant 
Command  (CCMD)  and  Service  assessments  completed  as  part 
of  the  Cybersecurity  Assessment  Program,  21  cybersecurity 
operational  test  and  evaluation  (OT&E)  events  of  acquisition 
systems,  and  continued  efforts  to  enhance  assessment  capabilities 
via  cyber-range  events.  During  this  year’s  CCMD  exercises  and 
acquisition  program  operational  tests,  cyber  Opposition  Forces 
(OPFOR)  portraying  adversaries  with  beginner  or  intermediate 
cyber  capabilities  were  able  to  demonstrate  that  many  DOD 
missions  are  currently  at  risk  from  cyber  adversaries.  CCMD 
and  Service  authorities  have  yet  to  consistently  show  that  critical 
missions  can  be  assured  in  scenarios  where  an  intermediate  or 
advanced  cyber  adversary  contests  these  missions. 

During  the  Turbo  Challenge  14  exercise,  a  combination  of 
skilled  local  defenders  and  security-conscious  network  users  and 
administrators  denied  the  Cyber  OPFOR’s  attempts  to  impact 
missions  on  the  U.S.  Transportation  Command’s  network.  This 
is  one  of  the  few  times  a  CCMD  quickly  detected  and  effectively 
responded  to  thwart  an  attack  by  an  intermediate-level  cyber 
adversary.  During  this  assessment,  the  U.S.  Transportation 
Command  demonstrated  the  following  key  security  tenets: 

•  Implementation  and  enforcement  of  strong  passwords  and 
password  storage  requirements 

•  Hardening  of  outward-facing  servers 

•  Consistent  review  of  network  logs  using  automated  scripts 

•  Effective  incident  response  and  reporting  processes. 

Notwithstanding  this  infrequent  success,  the  continued 
development  of  advanced  cyber  intrusion  techniques  makes  it 
likely  that  determined  cyber  adversaries  can  acquire  a  foothold 
in  most  DOD  networks,  and  could  be  in  a  position  to  degrade 
important  DOD  missions  when  and  if  they  chose  to.  It  is 
therefore  critical  that  DOD  network  defenders,  and  operators 
of  systems  residing  on  DOD  networks,  learn  to  ‘fight  through’ 
cyber  attacks,  just  as  they  are  trained  to  fight  through  more 
conventional,  ldnetic  attacks. 

DOD  continued  to  improve  compliance  with  policies  intended 
to  improve  cybersecurity,  such  as  ensuring  known  software 
patches  are  installed  on  time.  Consequently,  during  FY14 
DOT&E  assessments,  Red  Teams  report  that  some  beginner-  and 
intermediate-level  network  intrusion  exploits  did  not  work  as 
frequently  as  they  have  in  the  past.  However,  fundamental 
vulnerabilities  continue  to  persist  in  most  networks,  and 
processes  to  ensure  accountability  for  security  policy  violations 
have  not  matured. 

Exercise  authorities  permitted  more  realistic  OPFOR  activities 
on  operational  networks  in  FY14  than  during  previous  years, 
but  tended  to  limit  activities  to  acquiring  network  accesses 
and  exfiltration  of  information,  rather  than  more  disruptive 
activities  such  as  denial  of  service  attacks.  While  these  limits 


are  understandable  due  to  constraints  associated  with  operational 
networks,  operationally  realistic  assessments  require  realistic 
cyber  effects.  Without  realistic  cyber  effects,  the  training 
audience  may  have  a  false  sense  of  security  that  their  missions 
were  not  subject  to  degradation,  and  the  operators  and  network 
defenders  miss  the  opportunity  to  detect  and  respond  to  realistic 
cyber  attacks.  To  address  this  requirement,  DOD  needs  improved 
methods  and  range  environments  to  better  characterize  and 
simulate  cyber  effects  for  both  assessments  and  training. 

In  FY14,  DOT&E  began  examining  CCMDs’  ability  to  sustain 
critical  missions  when  subjected  to  realistic  cyber  threats.  These 
efforts  focus  on  missions  deemed  most  critical  by  the  CCMDs, 
and  will  help  increase  the  visibility  and  realism  of  cybersecurity 
assessments. 

DOD  initiatives  such  as  the  Joint  Information  Environment  (JIE) 
and  the  Cyber  Mission  Force  are  intended  to  address  some  of  the 
inherent  challenges  with  securing  DOD  networks.  DOT&E  will 
examine  the  effectiveness  of  new  Cyber  Protection  Teams  during 
future  assessments  with  CCMDs  and  Services,  and  will  fully  test 
the  JIE  as  it  is  implemented. 

During  FY14,  DOT&E  refocused  cybersecurity  assessments 
to  help  CCMDs  and  Services  reduce  the  number  of  persistent 
cybersecurity  vulnerabilities  that  DOT&E  has  reported  on  in 
previous  years.  Assessments  now  include  a  “fix”  phase  outside 
of  the  formal  assessment,  during  which  a  DOT&E-sponsored 
team  will  advise  CCMD  and  Service  personnel  on  the 
implications  of  existing  vulnerabilities,  ways  to  address  critical 
cybersecurity  vulnerabilities,  and  points  of  contact  for  further 
assistance. 

At  the  request  of  U.S.  Pacific  Command  and  several  other 
CCMDs,  DOT&E  has  begun  executing  more  frequent  cyber 
assessments,  including  “fix”  phases,  to  help  improve  the 
commands’  cybersecurity  posture  and  assess  the  impacts  of 
emerging  cyber  threats. 

During  FY14,  DOT&E  increased  the  interaction  between  cyber 
Red  Teams  and  network  defenders  following  assessments  to  help 
improve  defender  awareness  of  the  signs  and  optimal  responses 
to  cyber  intrusions.  DOT&E  sponsored  the  development 
of  ‘cyber  playbooks’  and  battle  drills  during  which  network 
defenders  can  practice  enhanced  tactics,  techniques,  and 
procedures  (TTPs). 

Realistic  cybersecurity  assessments  require  operationally 
representative  participation  by  network  defenders.  ‘Tier  2’ 
network  defenders,  which  provide  regional  network  defense, 
provide  critical  capabilities  that  augment  the  local  network 
defenders’  ability  to  detect  and  react  to  network  intrusions. 

During  FY14,  Tier  2  network  defenders  provided  more  active 
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support  to  DOT&E  assessments,  although  more  consistent  Tier  2 
involvement  is  required  in  the  future. 

In  FY14,  DOT&E  revised  and  published  procedures  for 
cybersecurity  OT&E  of  acquisition  programs,  providing  specific 
measures  and  standards  for  conducting  cybersecurity  tests. 
Cybersecurity  OT&E  will  continue  to  focus  on  identifying 
significant  cybersecurity  vulnerabilities,  and  characterizing  the 
impact  of  the  vulnerabilities  on  operational  missions.  DOT&E 
identified  critical  cybersecurity  vulnerabilities  in  most  of  the 
acquisition  programs  that  were  operationally  tested  during  FY14. 

During  FY14,  the  demand  for  resources  and  skilled  cybersecurity 
personnel  needed  to  support  operations,  training,  and  assessment 
increased  across  the  DOD.  Cyber  experts  were  needed  in  greater 
numbers  to  develop  cyber-secure  capabilities;  to  defend  networks 
and  systems;  to  provide  cyber  Red  Teams  to  support  training, 
assessments,  and  tests;  to  plan,  conduct,  and  analyze  tests  and 
assessments;  and  to  create  ranges  and  range  environments  to 
support  the  activities  discussed  in  this  section.  Demand  has 
begun  to  exceed  the  capacity  of  existing  personnel  able  to 
portray  cyber  threats,  and  projected  FY15  personnel  needs 
for  cybersecurity  tests  and  assessments,  as  well  as  training  for 
the  Cyber  Mission  Force  personnel  in  support  of  U.S.  Cyber 
Command,  may  not  be  met  unless  critical  resource  shortfalls  are 
addressed. 

During  FY14,  leadership  at  U.S.  Strategic  Command,  U.S. 

Cyber  Command,  and  U.S.  Pacific  Command  approved  Standing 
Ground  Rules  for  a  Persistent  Cyber  Opposing  Force  (PCO). 


These  ground  rules,  proposed  by  DOT&E,  permit  year-round 
operations  by  the  cyber  OPFOR  to  enable  a  more  representative 
portrayal  of  potential  cyber  adversaries.  U.S.  Northern 
Command  also  agreed  to  a  PCO  beginning  in  FY15.  The  PCO 
construct  will  allow  heavily-tasked  Red  Team  assets  to  support 
more  assessments  by  optimizing  Red  Team  targeting  boards  and 
aggressing  more  targets  throughout  the  year.  Results  of  the  PCO 
are  also  expected  to  help  set  initial  conditions  for  cybersecurity 
OT&E. 

To  improve  DOD’s  cybersecurity  posture,  DOT&E  recommends 
the  CCMDs  and  Services  do  the  following: 

•  Demonstrate  fight-through  capabilities  and  resiliency  for 
all  critical  missions;  these  demonstrations  should  include 
realistic  Cyber  OPFOR  play  and  active  involvement  by  Tier  2 
computer  network  defense  service  providers. 

•  Require  higher  levels  of  cybersecurity  accountability  for 
networks  and  systems  needed  for  critical  missions. 

•  Routinely  include  the  effects  of  a  representative  cyber  OPFOR 
in  training  exercises,  as  opposed  to  training  in  the  unlikely 
benign  cyber  environment. 

•  Emphasize  network  defense  fundamentals 

-  Implementation  and  enforcement  of  strong  passwords  and 
storage  requirements 

-  Hardening  of  outward-facing  servers 

-  Consistent  review  of  logs  at  all  tiers 

•  Exercise  and  improve  incident  response  and  reporting 
processes. 


FY1 4  ACTIVITIES 


Cybersecurity  Assessment  Program  Events 

In  FY14,  DOT&E,  in  conjunction  with  the  Army  Test  and 
Evaluation  Command;  the  Commander,  Operational  Test  and 
Evaluation  Force;  the  Marine  Corps  Operational  Test  and 
Evaluation  Activity;  the  Joint  Interoperability  Test  Command; 
and  the  Air  Force  Operational  Test  and  Evaluation  Center 
completed  15  cybersecurity  assessments.  The  assessments  were 
of  nine  CCMD  and  three  Service  exercises,  and  of  three  visits  to 
operational  sites  not  during  an  exercise  (see  Table  1). 

DOT&E’s  Cybersecurity  Assessment  Program  included  planning 
and  conduct  of  events,  both  during  large-scale  training  exercises 
and  at  operational  sites  during  events  other  than  a  training 
exercise.  DOT&E  also  conducted  Theater  Cyber  Readiness 
Campaign  (TCRC)  assessments,  which  comprised  a  series  of 
smaller  assessment  events  focused  on  specific  problems  and 
topics  of  interest  to  improve  cybersecurity.  These  sub-events 
assessed  vulnerabilities  identified  during  prior  assessments 
and  the  impacts  of  emerging  cyber  threats.  Each  TCRC 
phase  culminated  in  a  capstone  assessment  event — usually 
a  major  exercise — where  all  elements  of  the  TCRC  could 
be  simultaneously  assessed.  DOT&E  has  conducted  TCRC 
activities  at  three  CCMDs  to  date,  and  will  expand  these  efforts 
to  other  CCMDs  in  the  future. 


Persistent  Cyber  Opposing  Force  (PCO) 

During  FY14,  leadership  at  U.S.  Strategic  Command,  U.S. 

Cyber  Command,  and  U.S.  Pacific  Command  approved  Standing 
Ground  Rules,  proposed  by  DOT&E,  for  a  Persistent  Cyber 
OPFOR  (PCO).  The  rules  permit  year-round  operations  by 
the  cyber  OPFOR  to  enable  a  more  representative  portrayal  of 
potential  cyber  adversaries.  U.S.  Northern  Command  also  agreed 
to  a  PCO  beginning  in  FY15.  The  PCO  will  allow  heavily-tasked 
Red  Teams  to  support  more  assessments  by  optimizing  Red  Team 
targeting  boards  and  aggressing  more  targets  throughout  the  year. 
Results  of  the  PCO  are  also  expected  to  help  set  initial  conditions 
for  cybersecurity  OT&E  of  acquisition  programs. 

Although  the  PCO  construct  may — through  efficiencies — reduce 
the  OPFOR  workload  for  a  given  event,  these  efficiencies  are  not 
expected  to  offset  the  growth  in  demand  for  cyber  experts. 

Improvement  of  Cyber  Threat  Assessments 

DOT&E  has  partnered  with  multiple  DOD  organizations  to  form 
teams  possessing  cyber,  T&E,  cyber  range,  and  other  expertise  to 
support  cybersecurity  assessments,  including: 

Exercise  Support  Team.  The  Defense  Intelligence  Agency 
Exercise  Support  Team  developed  detailed  threat  folders  to 
improve  the  understanding  and  portrayal  of  cyber-adversary 
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capabilities,  and  also  supported  the  design  and  execution  of 
exercise  scenarios. 

Standing  Test,  Assessment,  and  Rehearsal  Team  (START). 

The  START  helped  ensure  the  right  talent  sets  were  integrated 
into  DOT&E-sponsored  assessment  activities.  In  FY14,  the 
START  supported  a  series  of  cyber-range  events  (Project  C)  that 
stressed  range  capabilities  and  environments,  while  also  affording 
new  Cyber  Protection  Teams  the  opportunity  to  defend  against 
cyber  attacks  on  realistic  networks.  U.S.  Cyber  Command 
partnered  with  DOT&E  on  these  cyber-range  events,  and  is 
employing  the  results,  which  included  a  draft  Cyber  Protection 
Team  (CPT)  tactics  guide,  to  help  identify  the  appropriate 
training  curriculum  for  the  68  CPTs,  refine  CPT  tactics,  and 
identify  metrics  to  assess  CPT  performance.  CPT  personnel 
were  appreciative  of  these  training  opportunities,  and  DOT&E 
will  continue  to  look  for  opportunities  to  engage  with  CPTs  and 
provide  CPT  assessment  results  to  U.S.  Cyber  Command. 

DOD  Enterprise  Cyber-Range  Environment  (DECRE).  The 

DECRE  continued  to  mature  its  cyber-range  capabilities,  but  at  a 
slower  pace  than  desired  by  U.S.  Cyber  Command,  the  training 
community,  and  the  Research,  Development,  Test,  and  Evaluation 
community.  Major  accomplishments  in  FY14  by  the  DECRE 
components  include: 

•  The  Test  Resources  Management  Center  (TRMC)  fielded  the 
cloud-based  Regional  Service  Delivery  Point  for  enhanced 
range  capability  and  connectivity. 

•  The  Joint  Staff  J6  created  several  cyber  environments  in  which 
to  examine  cyber  effects  not  suitable  for  operational  networks. 

•  The  TRMC’s  National  Cyber  Range  became  fully  operational 
and  is  now  looldng  at  ways  to  expand  capacity  to  meet  the 
growing  demand  for  range  events. 

All  of  these  DECRE  accomplishments  are  positive  and 
noteworthy,  but  the  demand  for  repeatable,  routine,  and 
distributed  events  exceeds  current  capabilities,  and  demand  is 
expected  to  increase  significantly  across  the  Future  Years  Defense 
Program. 

Partnerships  and  Collaboration.  Several  Research  and 
Development  organizations  have  made  existing  lab  environments 
available  and  performed  important  assessments  to  characterize 
the  effects  of  cyber  attacks.  Mission  areas  examined  included: 

•  Ballistic  Missile  Defense  -  DOT&E  partnered  with  the  Missile 
Defense  Agency  (MDA)  to  plan  and  execute  four  events  of 
increasing  complexity  and  realism  to  examine  potential  cyber 
vulnerabilities. 


•  Aegis  -  DOT&E  partnered  with  Navy  Red  Team,  Wallops 
Island  and  Dahlgren  test  facilities,  and  Combat  Direction 
Systems  Activity  (Dam  Neck)  to  characterize  and  understand 
vulnerabilities  focused  on  the  Aegis  Combat  Systems.  Events 
provided  information  on  the  scope  and  duration  of  cyber 
effects  to  inform  Program  Office  development. 

•  Command,  Control,  and  Intelligence  Systems  -  DOT&E 
partnered  with  the  Joint  Staff  J6  Command,  Control, 
Communications,  and  Computers  Assessment  Division  to 
create  an  environment  to  examine  cybersecurity  aspects  of 
the  common  operating  picture  and  situational  awareness 
systems.  Events  identified  and  characterized  cyber  effects  to 
be  introduced  into  training  exercises.  Continuing  efforts  will 
expand  the  systems  and  environment  to  explore  a  wider  variety 
of  cyber  effects. 

Both  the  MDA  and  the  Navy  have  identified  ways  to  improve 
cybersecurity  for  their  respective  programs  through  these 
assessment  activities. 

The  Naval  Postgraduate  School  developed  a  Malicious  Activity 
Simulation  Tool  (MAST),  which  is  ready  for  testing  in  realistic 
network  environments.  DOT&E  is  overseeing  the  efforts  to  test 
this  capability  on  a  cyber  range  to  confirm  readiness  to  support 
training  and  assessment  of  network  personnel. 

Several  National  Labs  (Sandia  National  Labs,  Johns  Hopkins 
Applied  Physics  Lab,  and  MIT  Lincoln  Labs)  delivered  or  are 
developing  prototypes  of  new  instrumentation  and  visualization 
capabilities,  new  products  for  traffic  generation,  and  new  ways 
to  automate  or  virtualize  network  environments  and  activities. 
These  new  capabilities  will  help  make  cyber- range  environments 
more  operationally  realistic,  and  will  also  help  optimize  the 
employment  of  range  capabilities  in  repeatable  and  distributed 
events. 

The  Army’s  Threat  Systems  Management  Office  (TSMO) 
played  a  leading  role  in  the  planning  and  execution  of  many 
DOT&E-sponsored  cyber-range  experiments,  identification  and 
acquisition  of  new  Red  Team  capabilities,  testing  and  fielding  of 
cyber-range  Regional  Service  Delivery  Points,  and  management 
and  operation  of  the  PCO.  TSMO  and  the  other  Service  Red 
Teams  continued  to  provide  Cyber  OPFOR  support  to  many  of 
the  FY14  exercise  assessments,  as  well  as  acquisition  testing. 
Other  Service  Red  Teams  also  provided  critical  support  in 
portraying  cyber  adversaries  in  exercise,  tests,  and  range  events. 
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TABLE  1.  CYBERSECURITY  ASSESSMENT  PROGRAM  EVENTS  IN  FY14 


EXERCISE 

AUTHORITY 

EVENT 

ASSESSMENT  AGENCY 

U  S.  Africa  Command 

Epic  Guardian  2014 

(exercise  cancelled,  conducted  as  Site  Visit) 

ATEC 

U.S.  Central  Command 

Site  Visit  -  Special  Operations  Command  Central 

ATEC 

U.S.  Cyber  Command 

Cyber  Flag  2014 

ATEC 

U.S.  European  Command 

No  Assessment  Opportunity 

ATEC 

U.S.  Northern  Command 

Vigilant  Shield  2014 

AFOTEC 

U.S.  Pacific  Command 

Cyber  Readiness  Campaign  Event  -  Physical  Security 

START 

Cyber  Readiness  Campaign  Event  -  Network  Hygiene 

Cyber  Readiness  Campaign  Event  -  Knowledge  Management 

U.S.  Southern  Command 

Site  Visit  -  Joint  Interagency  Task  Force  South 

ATEC 

U.S.  Special  Operations  Command 

Tempest  Wind  2014 

ATEC 

U.S.  Strategic  Command 

GlobalThunder2014 

JITC 

Global  Lightning  2014 

JITC 

U.S.  Transportation  Command 

Turbo  Challenge  2014 

JITC 

U.S.  Army 

Warfighter  2014-4 

ATEC 

U.S.  Navy 

Valiant  Shield  2014 

COTF 

U.S.  Air  Force 

No  Assessment  Opportunity 

AFOTEC 

U.S.  Marine  Corps 

Ulchi  Freedom  Guardian  2014 

MCOTEA 

AFOTEC  -  Air  Force  Operational  Test  and  Evaluation  Center  JITC  -  Joint  Interoperability  Test  Command 

ATEC  -  Army  Test  and  Evaluation  Command  MCOTEA  -  Marine  Corps  Operational  Test  and  Evaluation  Activity 

COTF  -  Commander,  Operational  Test  and  Evaluation  Force  START  -  Standing  Test,  Assessment,  and  Rehearsal  Team 


Cybersecurity  OT&E  of  Acquisition  Programs 

In  FY14,  DOT&E  approved  cybersecurity  test  plans  for 
82  Service  and  DOD  systems,  including  62  Test  and  Evaluation 
Master  Plans,  26  operational  test  plans,  and  25  related  test 
documents.  DOT&E  cybersecurity  subject  matter  experts 
observed  cybersecurity  tests  and  reviewed  test  data  for 
21  systems  across  the  warfare  domains. 

In  August  2014,  DOT&E  issued  updated  procedures  for  OT&E 
of  cybersecurity  in  acquisition  programs.  The  procedures  specify 
the  information  needed  for  planning,  conducting,  and  reporting 


cybersecurity  operational  testing  that  includes  a  cooperative 
vulnerability  and  penetration  assessment,  and  an  adversarial 
assessment.  The  purpose  of  the  cooperative  assessment  is 
to  identify  the  cybersecurity  vulnerabilities  of  a  system  in 
cooperation  with  the  program  manager  and  to  allow  the  program 
to  fix  them.  The  adversarial  assessment  then  evaluates  the  ability 
of  a  unit  equipped  with  the  system  to  support  assigned  missions 
in  the  expected  operational  environment  in  the  presence  of  a 
realistic  cyber  threat. 
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FINDINGS,  TRENDS,  AND  ANALYSIS 


Assessment  Structure 

FY14  continued  the  FY13  trend  of  fewer  exercises  available  or 
suitable  for  assessment;  the  impetus  for  this  in  FY13  was  the 
sequester,  and  reductions  in  exercise  funds  continued  into  FY14 
(see  Figure  1).  The  assessments  outside  of  an  exercise  reflect 
both  the  declining  number  of  large-scale  exercises  and  the 
implementation  of  focused  opportunities  to  find,  fix,  and  verify. 
Service-level  assessment  remained  at  a  level  of  one  per  Service. 


FY10  FY11  FY12  FY13  FY14 


Service  (Outside  Exercise)  Service  (During  Exercise) 

COCOM  (Outside  Exercise)  COCOM  (During  Exercise) 

Figure  1 .  Cybersecurity  Assessments  FY10-  FY14 

As  in  previous  years,  the  most  common  adversary  cyber 
activity  permitted  and  portrayed  during  FY14  exercises 
was  the  compromise  and  exfiltration  of  critical  operational 
information.  The  exercise  authority’s  desire  not  to  affect 
exercise  operations  usually  prevented  the  opposing  force  from 
using  the  compromised  information  to  influence  operations. 
These  limitations  minimized  the  value  of  cybersecurity  actions 
for  training  and  assessment.  In  several  exercises,  exercise 
authorities  allowed  Red  Teams  to  conduct  some  level  of  denial 
of  service  and  data  manipulation.  This  is  a  welcome  move 
towards  improving  threat  realism  and  the  creation  of  observable 
mission  effects. 

FY14  assessments  increasingly  included  active  participation 
of  the  local  network  defenders  and  the  next  higher  layer  of 
network  defense,  the  Tier  2  computer  network  defense  service 
providers.  The  increased  participation  is  a  notable  improvement 
over  that  observed  in  previous  years,  and  enables  assessment  of 
both  the  local/proactive  defenses  (standards  compliance,  patch 
management,  vulnerability  management)  and  the  defensive 
activities  conducted  at  higher  echelons  that  involve  the 
detection,  reaction,  and  response  to  cyber  threats.  Furthermore, 
realistic  participation  by  network  defenders  is  required  for 
an  adequate  assessment  of  the  ability  of  CCMDs  and  Service 
components  to  sustain  critical  missions  when  under  cyber 
attack. 


Assessment  Findings 

Red  Teams  portraying  a  Cyber  OPFOR  successfully  accessed 
target  networks  primarily  through  vulnerable  web  services 
and  social  engineering  (phishing).  Similar  to  FY13,  Red 
Teams  routinely  expanded  access  across  networks  using  stolen 
credentials.  The  asymmetric  nature  of  cyber  operations  allows 
even  a  single  default  or  weak  password  to  lead  to  rapid  access 
and  exploitation  of  the  network.  This  is  particularly  true  when 
the  password  belongs  to  an  individual  with  elevated  privileges. 
FY14  assessments  revealed  numerous  violations  of  DOD 
password  security  policies,  which  indicates  the  policies  are 
either  too  difficult  to  implement,  too  hard  to  enforce,  or  both. 

On  the  other  hand,  compliance  with  relevant  network  security 
standards  exceeded  85  percent  overall,  and  compared 
to  FY13,  was  higher  in  all  areas  except  security  design 
and  configuration,  and  identification  and  authentication 
(passwords).  Compliance  assessments  determine  whether 
network  defensive  measures  are  in  place.  The  generally  poor 
defensive  performance  against  dedicated  attacks  by  Red  Teams 
shows  that  a  network  is  only  as  secure  as  its  weakest  link. 
Unless  compliance  levels  approach  100  percent,  it  is  likely  a 
dedicated  cyber  adversary  will  succeed  in  accessing  a  network. 
Flence  it  is  critical  that  network  users  and  defenders  learn  to 
fight  through  and  accomplish  missions  in  the  face  of  network 
security  breaches. 

In  FY14,  certain  areas  of  network  defense  improved  over 
previous  years.  Regional  (Tier  2)  computer  network  defense 
service  providers,  which  provide  key  support  to  the  local 
defenders,  participated  in  half  of  the  assessed  exercises. 
Protective  defense,  in  the  forms  of  phishing  discovery  and 
perimeter  defense  configurations,  prevented  several  attempted 
Red  Team  incursions.  These  successes  reflect  improved 
personnel  awareness  to  recognize  and  report  phishing  emails, 
better  filters  for  identifying  and  blocking  phishing  emails, 
and  implementation  of  settings  to  block  common  intrusion 
techniques  used  in  these  emails.  Defenses  would  be  further 
improved  by  hardening  outward-facing  servers  and  limiting 
the  amount  of  sensitive  information  available  on  public 
portals. 

Some  network  defenders  demonstrated  the  ability  to  detect 
intrusions  by  reviewing  logs  of  network  and  sensor  activity, 
and  initiating  actions  to  counter  the  adversary  presence  on 
the  network.  In  over  half  of  the  FY14  assessments,  local 
network  defenders  initiated  these  detections  and  responses, 
and  coordinated  the  response  with  regional  computer  network 
defenders.  Such  coordinated  responses,  when  executed  well, 
can  protect  critical  mission  systems  from  cyber  attacks. 

In  many  cases,  however,  the  response  actions  were  not 
quick  enough  to  preclude  an  intermediate  or  advanced  cyber 
adversary  from  pivoting  to  another  foothold  or  escalating 
privileges  within  the  compromised  network.  Additionally, 
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some  responses  were  to  reboot  or  reload  software  for  systems 
believed  to  be  compromised  or  in  a  degraded  mode.  Depending 
on  the  operational  phase  of  the  exercise,  rebooting  or  reloading 
software  denies  users  mission-critical  services,  and  does 
not  contribute  to  the  commander’s  ability  to  fight  through  a 
cyber  attack.  Reloading  software  can  also  result  in  the  loss  of 
previously  installed  software  patches,  making  systems  more 
susceptible  to  cyber  attack. 

Although  many  of  the  elements  of  network  protection  and 
defense  were  observed  in  FY14  exercises,  the  lack  of  mature  and 
well-rehearsed  procedures  often  precluded  effective  integration  of 
network  defense  capabilities,  placing  missions  at  risk.  DOT&E 
assessed  that  at  least  one  mission  in  each  exercise  assessment  was 
at  high  risk  because  of  observed  cyber  activities,  including: 

•  Loss  of  operational  security  resulting  from  the  compromise  of 
sensitive  information 

•  Data  manipulation 

•  Denial  of  service 

Several  CCMDs  have  initiated  development  of  Cyber  Playbooks 
that  are  intended  to  achieve  more  accurate  and  timely  execution 
of  responses  to  cyber  attacks.  To  encourage  these  efforts,  and 
to  evaluate  their  effectiveness,  DOT&E  initiated  planning  with 
three  CCMDs  to  begin  a  focused  examination  of  the  CCMD’s 
ability  to  sustain  important  missions  when  subjected  to  realistic 
cyber  stress.  These  efforts  will  result  in  multi-year  Cyber 
Assessment  Master  Plans  (CAMPs)  centered  on  the  missions 
deemed  most  critical  by  the  CCMDs. 


Execution  of  CAMPs  will  support  implementation  of  the 
Chairman  of  the  Joint  Chiefs  of  Staff  Execute  Order,  published 
in  February  2011,  and  re-emphasized  by  the  Secretary  in 
December  2012,  which  required  routine  training  and  validation  of 
procedures  that  enable  execution  of  critical  missions  in  contested 
cyber  environments.  To  date,  DOT&E  has  yet  to  observe  a 
mission  demonstration  in  an  advanced  cyber-threat  environment. 

DOT&E  found  significant  vulnerabilities  on  nearly  every 
acquisition  program  that  underwent  cybersecurity  OT&E  in 
FY14.  Program  managers  worked  to  resolve  vulnerabilities 
found  from  cybersecurity  testing  in  prior  years,  but  FY14 
testing  revealed  new  vulnerabilities.  Corrections  to  past 
vulnerabilities  have  required  modifications  to  system 
architecture;  hardware,  firmware,  and  configurations;  system 
software;  training;  and  operational  procedures.  As  in  FY13, 
significant  vulnerabilities  found  during  OT&E  could  have  been 
found  and/or  remedied  during  earlier  phases  of  development. 
Nearly  all  the  vulnerabilities  were  discoverable  with  novice-  and 
intermediate-level  cyber  threat  techniques.  The  cyber  assessment 
teams  did  not  need  to  apply  advanced  cyber  threat  capabilities 
during  operational  testing. 

DOT&E  found  that  some  programs  had  not  adequately  planned 
for  cybersecurity  testing.  This  resulted  in  insufficient  time  to 
perform  adequate  cooperative  testing,  implement  fixes,  and 
achieve  successful  adversarial  testing  results,  ft  also  negatively 
impacted  the  ability  of  cyber  teams  to  plan  and  execute  their  test 
activities  across  different  programs. 


REPORTS 


For  the  Cybersecurity  Assessment  Program,  DOT&E  issued  an 
assessment  report  for  each  exercise  or  site  visit  that  discussed 
observations,  findings,  and  discovered  vulnerabilities.  DOT&E 
also  issued  separate  reports  to  DOD,  CCMD,  and  Service 
leadership  highlighting  high-priority  observations.  For  OT&E 
of  acquisition  programs,  DOT&E  reported  the  cybersecurity 
test  results  as  an  integrated  part  of  operational  effectiveness, 
suitability,  and  survivability. 

DOT&E  also  published  five  memoranda  of  findings  in  areas  of 
concern  in  FY14.  Finding  memoranda  detail  specific  problems 
that  need  senior  leadership  attention.  DOT&E  addressed  the 
finding  memoranda  to  the  responsible  leadership  for  action. 
DOT&E  will  evaluate  corrective  actions  in  future  assessments. 

New  finding  memoranda  published  in  FY14  were: 

•  Defense  Connect  Online  (Released  November  2013).  This 
was  a  follow-on  to  a  September  2010  finding  that  reported 
means  by  which  the  DOD  chat/collaboration  system  could 
be  compromised.  It  reported  on  new  findings  as  well  as  the 
efficacy  of  prior  remediation.  DISA  has  responded  to  this 
report  noting  corrections  that  will  be  made  to  the  system  in 
question. 

•  Host-Based  Security  System  (Released  April  2014).  This 
was  a  follow-on  to  an  October  2012  finding  that  reported 


shortfalls  in  how  the  DOD  network  security  tool  was  providing 
inventory  data.  It  reported  on  new  findings  of  how  the  tool 
could  be  exploited.  DISA/CIO  have  responded  to  this  report 
noting  the  actions  that  will  be  taken  to  correct  the  finding. 

•  Electronic  Security  of  Special  Handling  Documents 
(Released  April  2014).  This  finding  reported  shortfalls 
regarding  how  sensitive  Alternate  Control  Measure  programs 
were  being  handled  on  classified  networks.  The  Joint  Staff, 
DOD  CIO,  and  USD(I)  have  provided  a  coordinated  response 
describing  corrective  measures  that  have  or  will  be  taken  to 
address  this  finding. 

•  Shipboard  Datalinks  (Released  June  2014).  This  finding 
reported  on  issues  identified  with  off-ship  datalink  security. 

The  Navy  has  responded  with  specific  actions  that  are  being 
taken  to  address  the  finding. 

•  Assessment  of  DOD  Cybersecurity  during  Major  Combatant 
Command  and  Service  Exercises  and  Major  Program 
Acquisitions  (released  September  2014).  This  detailed  report 
provided  classified  observations  and  analysis  concerning 
common  vulnerabilities  and  issues  uncovered  during  major 
exercises  and  acquisition  tests.  No  response  was  required. 
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FY15  GOALS  AND  PLANS 


A  major  goal  of  the  Cybersecurity  Assessment  Program  in 
FY15  is  to  assist  the  CCMDs  and  Services  in  improving  their 
cybersecurity  postures  by  finding  cybersecurity  problems, 
providing  information  to  fix  problems,  and  verifying  the  status 
of  implemented  fixes  to  previously  discovered  problems.  An 
additional  goal  for  cybersecurity  OT&E  is  to  implement  the  new 
test  procedures  to  improve  rigor  and  consistency  of  cybersecurity 
testing  for  acquisition  programs. 

Specific  FY15  goals  include: 

•  Publish  finding  memoranda  to  recommend  solutions  to 
significant  cybersecurity  problems  that  could  have  an  impact 
on  DOD  missions. 

•  Include  a  “fix”  phase  in  each  of  the  planned  assessments  of 
nine  large-scale  training  exercises,  four  operational  site  visits 
outside  of  exercises,  and  four  cyber  readiness  campaigns 
having  multiple  events  (see  Table  2). 

•  Expand  the  Standing  Ground  Rule  authorities  for  PCO 
operations  to  additional  CCMDs. 


•  Ensure  availability  of  certified  and  properly  trained  and 
equipped  Red  Teams  to  provide  representative  Cyber  OPFOR 
support  to  OT&E  and  exercise  assessments. 

•  Improve  realism  of  the  cyber  threat  levels  and  effects 
portrayed  during  all  tests  and  assessments. 

•  Expand  DOD  cyber-range  environments  to  support 
demonstration  of  advanced  cyber  effects,  and  development  and 
verification  of  cybersecurity  solutions. 

•  Publish  a  Handbook  for  the  Cybersecurity  Assessment 
Program  to  update  the  procedures,  expectations,  and 
requirements  for  cybersecurity  assessments  of  CCMDs  and 
Services. 

•  Work  with  DOD  test  organizations  to  plan  more  robust 
cybersecurity  testing  during  OT&E,  including  participation  by 
cyber  defenders  and  the  creation  of  mission  effects. 

•  Provide  technical  recommendations  to  programs  and 
acquisitions  organizations  based  on  the  data  gathered  from 
cybersecurity  assessments  during  OT&E. 


TABLE  2.  CYBERSECURITY  ASSESSMENTS  PROPOSED  FOR  FY15 

EXERCISE 

AUTHORITY 

EVENT 

ASSESSMENT  AGENCY 

U  S.  Africa  Command 

Judicious  Response  2015 

ATEC 

U.S.  Central  Command 

Site  Visit  -  Air  Forces  Central  Command 

ATEC 

Site  Visit  -  Marine  Corps  Forces  Central  Command 

ATEC/MCOTEA 

U.S.  Cyber  Command 

Cyber  Guard  Exercises 

JITC 

U.S.  European  Command 

Austere  Challenge  2015 

ATEC 

Cyber  Readiness  Campaign  Events 

ATEC 

North  American  Aerospace 
Defense  Command/U.S.  Northern 
Command 

Vigilant  Shield  2015  Cyber  Readiness  Campaign  Events 

AFOTEC 

U.S.  Pacific  Command 

Cyber  Readiness  Campaign  Events 

START,  AFOTEC,  ATEC 

U.S.  Southern  Command 

Integrated  Advance  2015 

ATEC 

U.S.  Special  Operations  Command 

To  Be  Identified 

MCOTEA 

U.S.  Strategic  Command 

Global  Lightning  2015 

JITC 

Cyber  Readiness  Campaign  Events 

JITC 

U.S. Transportation  Command 

Turbo  Challenge  2015 

JITC 

U.S.  Army 

Warfighter  201 5-4 

ATEC 

U  S.  Navy 

Joint  Task  Force  Exercise  -  USS  Roosevelt 

COTF 

U.S.  Air  Force 

Site  Visit  -  U.S.  Pacific  Air  Forces 

AFOTEC 

U.S.  Marine  Corps 

Site  Visit  11  -  Marine  Expeditionary  Force 

MCOTEA 

AFOTEC  -  Air  Force  Operational  Test  and  Evaluation  Center  JITC  -  Joint  Interoperability  Test  Command 

ATEC  -  Army  Test  and  Evaluation  Command  MCOTEA  -  Marine  Corps  Operational  Test  and  Evaluation  Activity 

COTF  -  Commander,  Operational  Test  and  Evaluation  Force  START  -  Standing  Test,  Assessment,  and  Rehearsal  Team 
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Test  and  Evaluation  Resources 


Public  law  requires  DOT&E  to  assess  the  adequacy  of 
operational  and  live  fire  testing  conducted  for  programs  under 
oversight,  and  to  include  comments  and  recommendations  on 
resources  and  facilities  available  for  operational  testing  and 
evaluation  (OT&E)  and  on  levels  of  funding  made  available 
for  OT&E  activities.  DOT&E  monitors  and  reviews  DOD  and 
Service-level  strategic  plans,  investment  programs,  and  resource 
management  decisions  to  ensure  capabilities  necessary  for 
realistic  operational  tests  are  supported.  This  report  highlights 
general  areas  of  concern  in  testing  current  systems  and  discusses 
significant  issues,  DOT&E  recommendations,  and  T&E  resource 
and  infrastructure  needs  to  support  operational  and  live  fire 
testing.  FY14  focus  areas  include: 

•  Adjustments  to  DOT &E  FY 1 5  Budget  Request 

•  Army  Support  of  OT&E 

•  Operational  Test  Agency  Support  for  Missile  Defense  Testing 

•  Cyber  Warfare 

•  Joint  Strike  Fighter  (JSF)  Advanced  Electronic  Warfare  Test 
Resources 

•  Electronic  Warfare  for  Land  Combat 


•  Navy  Advanced  Electronic  Warfare  Test  Resources  and 
Environments 

•  Equipping  Self-Defense  Test  Ship  (SDTS)  for  Aegis  Combat 
System,  Air  and  Missile  Defense  Radar  (AMDR)  and  Evolved 
SeaSparrow  Missile  (ESSM)  Block  2  Operational  Testing 

•  Midti-Stage  Supersonic  Targets 

•  Fifth-Generation  Aerial  Target 

•  Torpedo  Surrogates  for  Operational  Testing  of  Anti-Submarine 
Warfare  (ASW)  Platforms  and  Systems 

•  Submarine  Surrogates  for  Operational  Testing  of  Lightweight 
and  Fleavyweight  Torpedoes 

•  Threat  Modeling  and  Simulation  (M&S)  to  Support  Aircraft 
Survivability  Equipment  (ASE)  Testing 

•  Foreign  Materiel  Acquisition  Support  for  T&E 

•  Real-Time  Casualty  Assessment  (RTCA) 

•  Joint  Urban  Test  Capability  (JUTC) 

•  Hypersonic  Weapons  Test  Infrastructure 

•  Range  Sustainability 

•  Continuing  Radio  Frequency  Spectrum  Concerns 


Adjustments  to  DOT&E  FY1 5  Budget  Request 

Action  by  the  House  Armed  Services  Committee  (HASC), 
the  Senate  Armed  Services  Committee  (SASC),  the  House 
Appropriations  Committee,  and  the  Senate  Appropriations 
Committee  on  the  FY15  budget  request  included: 

•  HASC  and  SASC  approval  of  the  President’s  Budget  request 
in  the  FY15  National  Defense  Authorization  Act. 

•  Appropriations  increases  for  - 

-  Cyber  force  training  and  resiliency  ($10.0  Million) 

-  U.S.  Pacific  Command  cyber  range  training  ($4.9  Million) 

-  Cyber  Red  Team  and  training  ($3.8  Million) 

-  Threat  Resources  Analysis  ($5.0  Million) 

-  Joint  T &E  ($18.0  Million) 

Army  Support  of  Operational  Test  and  Evaluation 

For  the  fifth  year  in  a  row,  the  Army  is  reducing  the  funding 
level  for  the  direct  support  of  OT&E  of  Army  programs.  In 
FY10,  the  Army’s  budget  for  “Support  of  Operational  Testing” 
(PE  0605712A)  was  ~$78.4  Million,  and  as  of  FY15,  that  budget 
is  ~$49.2  Million  (FY10  $45.4  Million),  a  42  percent  reduction 
from  FY10  funding  levels.  In  FY10,  the  Army’s  budget  for  the 
“Army  Evaluation  Center”  (PE  06057 16A)  was  ~$63.9  Million, 
and  as  of  FY15,  that  budget  is  ~$55.0  Million  (FY10  $50.8 
Million),  a  21  percent  reduction  from  FY10  funding  levels. 

These  cuts  have  resulted  in  staff  level  reductions  in  both 
the  Army  Operational  Test  Command  (OTC)  and  the  Army 
Evaluation  Center  (AEC)  of  approximately  22  and  25  percent, 


respectively,  from  FY  10  to  FY14.  Further  cuts  in  staff  of 
10  and  6  percent,  respectively,  are  anticipated  in  FY16.  These 
reduced  staff  levels  are  likely  to  cause  delays  to  developmental 
and  operational  testing,  the  inability  to  conduct  simultaneous 
operational  test  events,  and  longer  timelines  for  the  release  of 
test  reports.  Delays  in  test  execution  and  test  reporting  may 
delay  acquisition  decisions.  The  small  savings  generated  by 
further  reducing  the  staff  of  OTC  and  AEC  may  result  in  a  cost 
penalty  to  acquisition  programs  that  is  proportional  to  spend 
rates  multiplied  by  the  duration  of  delay.  Other  smaller  but  still 
valuable  programs  may  be  delayed  even  longer,  as  priority  will 
be  given  to  the  Major  Defense  Acquisition  Programs. 

These  reductions  to  the  Army  T&E  operational  accounts  are 
part  of  broader  cuts  that  the  Army  has  taken  across  the  T&E 
enterprise,  including  in  the  office  of  the  Army  T&E  Executive. 
The  Army  T&E  Executive  performs  various  critical  roles 
managing  the  Army  T&E  enterprise  and  ensuring  T&E  adequacy 
within  the  Army  and  Chemical  and  Biological  Defense  Program 
(CBDP),  including  the  following: 

•  Establishing,  reviewing,  and  enforcing  Army  and  CBDP  T&E 
policy  and  procedures. 

•  Coordinating  and  facilitating  communication  with  OSD  on  all 
T&E  matters. 

•  Providing  oversight  and  policy  for  the  management  and 
operation  of  the  Headquarters,  Department  of  the  Army 
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(HQDA)  Major  Range  and  Test  Facility  Base  activities  and 
major  investments.  A  roughly  $1  Billion/year-enterprise. 

•  Managing  the  staffing  and  approval  process  for  Army  Test 
and  Evaluation  Master  Plans  that  require  HQDA  and  OSD 
approval. 

•  Supporting  the  Vice  Chief  of  Staff  Army  by  serving  as  a 
member  of  the  Board  of  Directors  Executive  Secretariat. 

•  Administering  the  Army  portion  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics’  Central 
Test  and  Evaluation  Investment  Program  and  Resource 
Enhancement  Program,  and  provide  representation  on  the  OSD 
Test  Investment  Coordinating  Committee. 

•  Ensuring  that  threat-representative  targets  and  threat 
simulators  are  validated  to  support  accreditation  for  test. 

In  2008,  the  Army  eliminated  the  office  of  the  Director,  Test 
and  Evaluation  Management  Agency  (TEMA)  within  the  Office 
of  the  Chief  of  Staff  of  the  Army,  and  moved  many  of  TEMA’s 
responsibilities  under  the  Army  T&E  Executive.  The  duties 
of  these  two  offices  are  now  being  performed  by  a  staff  of 
11  individuals,  with  an  additional  12  individuals  dedicated  to  the 
CBDP.  This  is  a  nearly  50  percent  reduction  over  the  past  5  years 
and  staffing  levels  are  now  such  that  the  ability  of  the  Office  to 
function  effectively  is  at  risk. 

In  a  memorandum  dated  November  12,  2014,  DOT&E 
recommended  the  Secretary  of  the  Army  reverse  these  trends.  In 
particular,  DOT&E  recommends  that  the  Army  restore  budgets 
that  will  maintain  FY14  staffing  levels  at  OTC  and  AEC,  as 
well  as  assure  staffing  levels  of  the  Army  T&E  Executive  are 
consistent  with  its  mission. 

Operational  Test  Agency  Support  for  Missile  Defense  Testing 

The  Ballistic  Missile  Defense  System  (BMDS)  Operational 
Test  Agency  (OTA)  is  customer- funded  by  the  Missile  Defense 
Agency  (MDA).  The  BMDS  OTA’s  mission  includes  test 
planning  and  execution;  system  evaluation,  analysis,  and 
assessment;  and  system-level  Modeling  and  Simulation  (M&S) 
accreditation  across  the  entire  MDA.  However,  all  BMDS  OTA 
funding  is  channeled  through  the  Test  Directorate.  Because  of 
this  structure,  the  BMDS  OTA  budget  has  suffered  percentage 
cuts  proportional  to  MDA  test  program  budget  reductions  over 
the  past  two  years.  Further,  as  Congress  has  restored  funding 
to  the  MDA  test  program,  the  BMDS  OTA  funding  remained 
constant.  Over  the  last  two  years,  the  BMDS  OTA  budget 
has  been  reduced  $3.6  Million  per  year  from  its  requested 
$16.1  Million  per  year,  resulting  in  significant  staff  reductions. 
The  Test  Directorate  recently  reduced  the  BMDS  OTA’s  FY15 
budget  an  additional  $1.0  Million  to  a  total  of  $11.5  Million  per 
year,  and  additional  reductions  are  anticipated  throughout  the 
Future  Years  Defense  Program.  These  cuts  are  not  consistent 
with  the  BMDS  OTA’s  mandate  and  have  resulted  in  the 
BMDS  OTA  operating  at  risk  in  critical  mission  areas  such  as 
system-level  M&S  accreditation  and  Ground-based  Midcourse 
Defense  operational  test  and  evaluation.  The  currently  unfunded 
requirement  for  these  two  areas  alone  require  an  additional 


$1.3  Million  per  year  funding  for  M&S  accreditation  and  an 
additional  $3.8  Million  per  year  funding  for  Ground-based 
Midcourse  Defense  operational  test  and  evaluation.  DOT&E 
strongly  recommends  that  the  BMDS  OTA  funding  line  be 
1)  realigned  at  the  agency  level  within  MDA,  and  2)  restored  to  a 
level  of  funding  appropriate  to  its  entire  mission. 

Cyber  Warfare 

Experimentation,  development,  testing,  training,  and  mission 
rehearsal  of  offensive  and  defensive  cyber-warfighting 
capabilities  require  representative  cyber  environments.  Such 
environments  are  created  with  distributed  cyber  ranges  and  labs 
that  provide  or  host  realistic  network  environments;  emulation 
of  adversary  targets  and  offensive/defensive  capabilities;  and 
representative  warfighter  systems,  network  defenses,  and 
operators.  Cyber  ranges  and  environments  can  be  joined  with 
other  DOD  ranges  as  critical  enablers  of  operations  in  the  air, 
land,  sea,  and  space  domains. 

In  FY11  and  FY12,  DOT&E  proposed  enhancements  to 
existing  facilities  to  create  the  DOD  Enterprise  Cyber  Range 
Environment  (DECRE)  comprised  of  the  National  Cyber  Range 
(NCR),  the  DOD  Cybersecurity  Range,  the  Joint  Information 
Operations  Range  (JIOR),  and  the  Joint  Staff  J-6  Command, 
Control,  Communications,  and  Computers  (C4)  Assessments 
Division  (C4AD).  Approved  enhancements  include  an  additional 
$172  Million  and  10  civilian  positions  for  the  DECRE,  and  are 
intended  to  provide: 

•  Consistent  portrayal  of  operationally  realistic,  threat- 
representative  cyber  environments 

•  Expansion  of  JIOR  operations  capacity  to  plan  and  rigorously 
execute  approximately  100  distinct  events  per  year 

•  Upgrades  to  introduce  cloud-based  Regional  Service  Delivery 
Points 

•  Incorporation  of  technologies  emerging  from  the  NCR  for 
rapid  design,  reconfiguration,  and  sanitization  of  networks 

•  Incorporation  of  various  Live,  Virtual,  and  Constructive 
capabilities 

•  Range  environments  where  advanced  cyber-attacks  can  be 
conducted  to  understand  the  scope  and  duration  of  cyber 
effects,  and  where  training  and  tactics,  techniques,  and 
procedures  development  and  validation  can  be  performed 

•  Archival  capabilities  to  record  and  play  back  live  events,  and 
blend  mixes  of  live  and  previously-recorded  events 

The  four  elements  of  the  DECRE  received  the  first  increment 
of  new  funding  in  FY14,  and  have  begun  to  reverse  some  of 
the  negative  trends  that  motivated  DOT&E  to  propose  these 
enhancements.  With  assistance  from  DOT&E,  the  C4AD  Team 
developed  a  high-fidelity  environment  to  examine  the  effects  of 
cyber-attacks  on  systems  that  support  Combatant  Commands’ 
(CCMD)  Common  Operating  Picture.  This  environment  has 
performed  multiple  demonstrations  to  characterize  potential 
cyber  effects  in  this  mission  area,  and  several  assessment  teams 
for  FY15  CCMD  exercises  will  employ  this  environment  to 
increase  the  threat  realism  of  their  assessments.  C4AD  is 
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growing  the  operational  and  network  realism  of  this  and  several 
other  environments  that  they  host. 

Other  environments  in  use  or  in  development  include  those 
for  missile  defense,  satellite  systems,  and  remote  testing  of 
interoperability  and  cyber  security  by  acquisition  programs 
via  connection  to  the  DECRE.  This  last  environment  will 
permit  a  program  manager  to  routinely  access  the  archives  of 
information-exchange  requirements  to  confirm  interoperability, 
subject  system  software  to  known  cyber-attacks,  and  receive 
automated  reports  of  the  cybersecurity  and  interoperability  status 
of  the  system  software.  C4AD  and  the  T&E  community  will  test 
this  new  environment  in  2QFY15,  and  it  should  be  accessible  via 
the  DECRE  by  the  end  of  FY15. 

The  NCR  experienced  a  substantial  increase  in  customers  in 
FY14,  and  needs  to  develop  options  for  expanding  significant 
NCR  capabilities  and  making  these  accessible  to  a  growing 
customer  base.  The  Test  Resource  Management  Center  (TRMC), 
which  oversees  the  NCR,  has  initiated  studies  to  examine  new 
capabilities  to  further  expedite  the  planning,  execution,  and 
sanitization  of  NCR  events. 

The  JIOR  initiated  an  upgrade  of  its  nearly  100  Service  Delivery 
Points  (SDPs)  with  the  new  Pico  SDP,  and  plans  to  migrate  to 
a  new  capability  set  that  will  interoperate  fully  with  the  new 
capability  under  development  by  the  TRMC’s  Joint  Mission 
Environment  Test  Capability  (JMETC)  Program  (i.e.,  both 
JIOR  and  JMETC  are  migrating  to  a  new  set  of  interoperability 
standards  that  define  the  future  DECRE).  These  migrations 
will  provide  efficiencies  for  the  DOD,  and  will  be  essential 
to  maximize  the  utility  of  the  next-generation  Regional  SDP 
(RSDP)  technology.  The  TRMC  has  completed  assembly  of  the 
third  RSDP  and  is  resourced  to  build  and  deliver  a  new  RSDP 
each  year  across  the  Future  Years  Defense  Program.  RSDPs 
are  effectively  cloud-based  mini-ranges  that  can  host  virtual 
environments,  instrumentation,  and  traffic  generation  capabilities, 
and  connect  to  other  nodes  or  RSDPs  via  the  JIOR  or  JMETC. 

As  funding  pennits,  DOT&E  will  initiate  development  of 
several  additional  environments  each  year,  often  driven  by  the 
need  to  characterize  cyber  effects  that  are  not  permissible  on 
operational  networks.  DOT&E  expects  that  these  high-fidelity 
cyber  environments  will  become  essential  to  cybersecurity  and 
interoperability  assessments,  OT&E,  and  also  to  the  training 
of  the  DOD  Cyber  Mission  Force  being  implemented  by  U.S. 
Cyber  Command. 

Although  many  improvements  are  in  progress,  DOT&E  expects 
the  demand  for  high-fidelity  cyber  environments  and  range  events 
will  continue  to  outpace  the  nascent  DECRE  capabilities.  For 
example,  U.S.  Cyber  Command  alone  estimates  that  the  Cyber 
Mission  Force  will  require  more  than  100  training  activities 
each  month,  a  great  deal  more  than  the  current  capability  for 
100  events  per  year  across  all  DOD  customers.  DOT&E  projects 
that  by  FY19,  160  range  events  will  be  needed  annually  to 
support  OT&E  for  oversight  programs,  and  many  more  by  the 
Services  for  non-oversight  programs.  DOT&E  also  projects 
that  in  FY15,  approximately  50  range  events  will  be  needed  for 


various  non-OT&E  events  that  DOT&E  will  support  on  behalf  of 
CCMDs  and  other  partner  organizations. 

The  integration  of  key  U.S.  and  coalition  range  nodes  and 
laboratories  for  distributed,  secure,  operationally  realistic,  and 
threat-representative  cyber  environments  will  further  expand  the 
demand.  DOT&E  will  continue  to  monitor  and  report  on  the 
evolution  of  DECRE  during  FY15.  DOT&E  recommends  that 
the  currently  fragmented  management  and  resourcing  of  DECRE 
be  consolidated  under  an  Executive  Agent  with  the  authority  to 
identify  requirements,  standards,  and  priorities  across  DECRE 
elements. 

Joint  Strike  Fighter  (JSF)  Advanced  Electronic  Warfare  Test 

Since  February  2012,  when  DOT&E  identified  shortfalls  in 
electronic  warfare  test  resources,  progress  in  procuring  these 
assets  has  been  very  slow.  These  shortfalls  prevent  development, 
testing,  and  timely  fielding  of  U.S.  systems  capable  of  operating 
successfully  against  threats  that  currently  exist,  are  proliferating, 
and  are  undergoing  an  accelerating  pace  of  significant  upgrades. 
While  FY13-18  funding  was  identified  to  address  these  shortfalls, 
and  this  was  updated  in  the  FY15-19  budget,  the  delay  in  acting 
to  use  the  funding  is  jeopardizing  the  opportunity  to  make  the 
needed  test  resources  available  in  time  to  support  developmental 
and  operational  testing  of  systems,  including  the  JSF. 

Capabilities  under  development  in  JSF,  F-22  Increment  3.2  A/B, 
B-2  Defensive  Management  System,  Long-Range  Strike  Bomber, 
Next  Generation  Jammer  for  the  EA-18G,  Countermeasures 
upgrades,  as  well  as  several  other  programs,  require  the 
combination  of  improved  government-owned  anechoic  chambers 
and  new  open-air  range  test  assets  recommended  by  DOT&E. 
DOT&E  recommendations  and  current  statuses  are  shown  in  the 
table  below. 


TABLE  1.  RECOMMENDATIONS  ON  ELECTRONIC  WARFARETEST  RESOURCES 

DOT&E  Recommendation 

Current  Status 

Developing  a  combination  of 
open-  and  closed-loop  threat  radar 
simulators  in  the  numbers  required 
for  operationally  realistic  open-air 
range  testing  of  JSF  and  other  systems 
beginning  in  2018. 

Risk  reduction  efforts  have  begun; 
however,  the  opportunity  to  procure 
the  number  and  type  of  systems 
needed  to  represent  the  threat  before 
2018  is  fleeting. 

Upgrading  the  government  anechoic 
chambers  with  adequate  numbers  of 
signal  generators  for  realistic  threat 
density. 

Initial  studies  of  materiel  solutions  to 
achieve  realistic  densities  have  begun 

•  The  Navy  chamber  has  procured 
initial  test  support  equipment 
for  direct  injection  capability  and 
executed  a  limited  F-35  electronic 
warfare  test  in  September  2014. 

•  The  JSF  program  has  yet  to  develop 
plans  to  integrate  chamber  testing 
into  the  verification  test  strategy. 

Upgrading  the  JSF  mission  data  file 
reprogramming  lab  to  include  realistic 
threats  in  realistic  numbers. 

An  initial  study  to  determine  upgrade 
requirements  has  begun. 

Providing  Integrated  Evaluation 
and  Analysis  of  Multiple  Sources 
intelligence  products  needed  to  guide 
threat  simulations. 

Products  have  been  completed  and 
delivered,  additional  requests  for 
information  have  been  submitted. 
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Regarding  the  shortfall  with  respect  to  JSF  testing  and  the 
time  lost  so  far,  the  challenges  to  delivering  the  desired  test 
environment  so  as  to  verify  performance  in  the  2018  Initial 
Operational  Test  and  Evaluation  (IOT&E)  can  be  largely 
overcome  with  efficient  and  aggressive  use  of  the  available 
funds.  The  risk  of  not  doing  so  is  two-fold:  a)  the  JSF  IOT&E 
of  Block  3F  capability  will  not  be  adequate — performance 
in  the  existing  threat  environment  will  not  be  known;  b)  the 
development  environment  sufficient  and  necessary  for  Block  4 
will  be  late  to  meet  the  need. 

Electronic  Warfare  for  Land  Combat 

Networked  mission  command  systems  that  support  the 
commander’s  mission  execution  across  the  Brigade  Combat 
Team  (BCT)  are  a  cornerstone  of  the  Army’s  modernization 
plan.  These  integrated  network  capabilities  are  distributed 
throughout  a  combat  formation  and  its  support  elements,  from 
the  brigade  command  posts  down  to  the  individual  dismounted 
Soldier.  Commanders  using  tactical  network  systems  have  the 
unprecedented  ability  to  transfer  information  such  as  voice, 
video,  text,  position  location  information,  and  high-resolution 
photographs  throughout  the  BCT,  and  provide  individual  Soldiers 
access  to  information  needed  to  complete  their  mission.  The 
expanded  use  of  these  radio  frequency  datalink-based  systems 
also  exposes  the  BCT  to  new  electronic  warfare  threat  vectors  the 
enemy  may  utilize. 

While  the  Army  Threat  Systems  Management  Office  has 
continued  to  improve  their  threat  electronic  warfare  simulator 
equipment,  it  has  not  kept  pace  with  the  advances  in  the  tactical 
network  systems  or  the  known  threat  capabilities  such  as 
advanced  jamming  and  direction-finding  techniques.  As  the 
Army  becomes  more  dependent  on  these  sophisticated  network 
technologies,  it  is  critical  that  the  developmental/operational 
test  communities  continue  to  identify  vulnerabilities  of  these 
systems.  Decision  makers  must  understand  these  inherent 
vulnerabilities,  as  well  as  the  ways  in  which  an  enemy  may 
choose  to  exploit  and/or  degrade  the  network.  These  critical 
threat  capabilities  are  needed  to  support  testing  of  Warfighter 
Information  Network  -  Tactical  Increment  2,  Nett  Warrior, 
Mid-Tier  Networking  Vehicular  Radio,  Manpack  radio,  and 
Joint  Battle  Command  -  Platform.  DOT&E  recommends  the 
Army  make  additional  resources  available  to  improve  and  expand 
its  ground-based  threat  electronic  warfare  capabilities  to  support 
operational  testing. 

Navy  Advanced  Electronic  Warfare  Test  Resources  and 
Environments 

Capability  for  Realistic  Representation  of  Multiple  Anti-Ship 
Cruise  Missile  (ASCM)  Seekers  for  Surface  Electronic  Warfare 
Improvement  Program  (SEWIP)  Operational  Testing 

This  gap  in  test  capability  was  identified  in  DOT&E ’s 
FY13  Annual  Report  as  “Additional  Electronic  Warfare 
Simulator  Units  for  Surface  Electronic  Warfare  Improvement 
Program  (SEWIP)  Operational  Testing.”  The  Navy  addressed  it 
with  development  of  a  programmable  seeker  simulator  that  could 


represent  different  ASCM  seekers  by  specifying  the  electronic 
waveform  emission  characteristics  for  one  of  several  possible 
threats.  The  effective  radiated  power  (ERP)  was  not  among  those 
characteristics,  with  the  result  that  simulated  attacks  by  ASCM 
representations  displayed  disparate  levels  of  ERP  that  were 
unlikely  to  be  encountered  during  a  stream  raid  attack  of  two 
ASCMs  (along  the  same  bearing  and  elevation  and  within  close 
proximity  of  one  another).  The  programmable  seeker  simulator, 
termed  the  “Complex  Arbitrary  Waveform  Synthesizer,”  needs 
to  be  modified  such  that  its  ERP  more  realistically  represents  the 
second  ASCM  of  a  dual  ASCM  stream  raid. 

The  next  SEWIP  Block  2  OT&E  is  projected  for  FY19.  This  is 
to  be  followed  by  Follow-On  Operational  Test  and  Evaluation 
(FOT&E)  on  a  Product  Line  Architecture-compliant  DDG  51 
with  Block  2  actually  integrated  with  the  Aegis  Combat 
System.  This  integration  was  not  part  of  the  Block  2  IOT&E. 
Subsequent  FOT&E  would  be  with  the  DDG  1000  and  CVN-78 
combat  systems.  Estimated  cost  to  add  the  ERP  improvement  is 
$5.0  Million. 

Long-Term  Improvement  in  Fidelity  of  ASCM  Seeker/Autopilot 
Simulators  for  Electronic  Warfare  Testing 

This  gap  in  test  capability  was  identified  in  DOT&E ’s  FY13 
Annual  Report  due  to  the  continued  reliance  on  manned 
aircraft  for  captive-carry  of  the  ASCM  seeker  simulators.  Such 
simulators  will  be  unable  to  demonstrate  kinematic  response  to 
electronic  attack  by  SEWIP  Block  3.  The  manned  aircraft  fly 
too  high  and  too  slow  for  credible  ASCM  representation  and 
are  unable  to  represent  ASCM  maneuvers.  Credible  ASCM 
representation  requires  a  vehicle  that  can  fly  at  subsonic  ASCM 
speeds  and  lower  altitudes  than  the  current  Lear  Jets;  can  home 
on  a  platform  representing  a  SEWIP  Block  3 -mounted  ship, 
using  a  threat-representative  radar  seeker  and  autopilot;  and  can 
respond  realistically  to  Block  3  electronic  jamming.  Plausible 
approaches  might  include: 

•  Recoverable,  unmanned  aerial  vehicles  using  embedded, 
miniaturized  simulators  that  are  maneuverable  at  ASCM 
speeds  and  altitudes 

•  Encrypted  telemetry  to  track  seeker/autopilot  responses  to 
electronic  attack 

•  Human-controlled  override  capability 

•  Use  of  an  unmanned,  remotely  controllable  Self-Defense  Test 
Ship  (SDTS)  that  would  tow  a  ship  target  for  the  unmanned 
aerial  vehicles  to  home  on.  SEWIP  Block  3  would  be  mounted 
on  the  SDTS,  as  would  hard-kill  elements  (missile  and  gun 
systems)  such  that  the  integrated  hard-kilPsoft-kill  (Block  3) 
combat  system  could  demonstrate  capability.  Currently, 

such  testing  is  at  the  combat  system  element  level,  leaving 
integrated  combat  system  capability  unknown. 

SEWIP  Block  3  IOT&E  is  projected  for  FY19.  FOT&E  of 
Block  3  integrated  with  the  DDG  1000  combat  system,  as  well 
as  FOT&E  with  the  CVN-78  combat  system,  should  occur 
subsequent  to  the  IOT&E.  Estimated  costs  are  $120.0  Million 
for  development,  testing,  and  acquisition.  Estimated  unit  cost  is 
$15.0  Million. 
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Equipping  Self-Defense  Test  Ship  (SDTS)  for  Aegis  Combat 
System,  Air  and  Missile  Defense  Radar  (AMDR)  and  Evolved 
SeaSparrow  Missile  (ESSM)  Block  2  Operational  Testing 

The  close-in  ship  self-defense  battle  space  is  complex  and 
presents  a  number  of  challenges  for  OT&E.  For  example,  this 
environment  requires: 

•  Weapon  scheduling  with  very  little  time  for  engagement 

•  AMDR  and  Close-In  Weapons  System  (CIWS)  (to  deal  with 
debris  fields  due  to  previous  successful  engagements  of 
individual  ASCMs  within  a  multi-ASCM  raid) 

•  Rapid  multi-salvo  kill  assessments  for  multiple  targets 

•  Transitions  from  ESSM  Command  Midcourse  Guidance  mode 
to  Home-All-the-Way  guidance  mode 

•  Conducting  BMD  and  area  air  defense  missions  (i.e., 
integrated  air  and  missile  defense)  while  simultaneously 
conducting  ship  self-defense 

•  Contending  with  stream  raids  of  multiple  ASCMs  attacking 
along  the  same  bearing,  in  which  directors  illuminate  multiple 
targets  (especially  true  for  maneuvering  threats) 

•  Designating  targets  for  destruction  very  close-in  by  CIWS 

Multiple  hard-kill  weapons  systems  operate  close-in,  including 
the  Standard  Missile  2  (SM-2),  the  ESSM,  and  the  CIWS. 

Soft-kill  systems  such  as  Nulka  Mk  53  decoy  launching  system 
also  operate  close-in.  The  short  timelines  required  to  conduct 
successful  ship  self-defense  place  great  stress  on  combat  system 
logic,  combat  system  element  synchronization,  combat  system 
integration,  and  end-to-end  performance. 

Navy  range  safety  restrictions  prohibit  close-in  testing  on  a 
manned  ship  because  the  targets  and  debris  from  successful 
intercepts  will  pose  an  unacceptable  risk  to  the  ship  and  personnel 
at  the  ranges  where  these  self-defense  engagements  take  place. 
These  restrictions  were  imposed  following  a  February  1983 
incident  on  the  USS  Antrim  (FFG  20),  which  was  struck  with  a 
subsonic  BQM-74  aerial  target  during  a  test  of  its  self-defense 
weapon  systems,  killing  a  civilian  instructor.  The  first  unmanned, 
remotely  controlled  SDTS  (the  ex-Stoddard)  was  put  into  service 
that  same  year.  A  similar  incident  occurred  in  November  2013, 
where  two  sailors  were  injured  when  the  same  type  of  aerial 
target  struck  the  USS  Chancellorsville  (CG  62)  during  what  was 
considered  to  be  a  low-risk  test  of  its  combat  system.  This  latest 
incident  underscores  the  inherent  dangers  of  testing  with  manned 
ships  in  the  close-in  battlespace. 

While  the  investigation  into  the  Chancellorsville  incident  has 
caused  the  Navy  to  rethink  how  they  will  employ  subsonic 
and  supersonic  aerial  targets  near  manned  ships,  the  Navy  has 
always  considered  supersonic  ASCM  targets  a  high  risk  to  safety 
and  will  not  permit  flying  them  directly  at  a  manned  ship.  The 
Navy  has  invested  in  a  current  at-sea,  unmanned,  remotely- 
controlled  test  asset  (the  SDTS)  and  is  using  it  to  overcome 
these  safety  restrictions.  The  Navy  is  accrediting  a  high-fidelity 
M&S  capability  utilizing  data  from  the  SDTS,  as  well  as  data 
from  manned  ship  testing,  so  that  a  full  assessment  of  ship 
self-defense  capabilities  of  non- Aegis  ships  can  be  completely 
and  affordably  conducted.  While  the  Navy  recognizes  the 
capability  as  integral  to  the  test  programs  for  certain  weapons 


systems  (the  Ship  Self-Defense  System,  Rolling  Airframe 
Missile  Block  2,  and  ESSM  Block  1)  and  ship  classes  (LPD-17, 
LHA-6,  Littoral  Combat  Ship,  LSD  41/49,  DDG  1000,  and 
CVN-78),  they  have  not  made  a  similar  investment  in  an  SDTS 
equipped  with  an  Aegis  Combat  System,  AMDR,  and  ESSM 
Block  2  for  adequate  operational  testing  of  the  DDG  51  Flight 
III  Destroyer  self-defense  capabilities.  The  current  SDTS  lacks 
the  appropriate  sensors  and  other  combat  system  elements  to  test 
these  capabilities. 

On  September  10,  2014,  DOT&E  issued  a  classified 
memorandum  to  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  with  a  review  of  the  Design  of 
Experiments  study  by  the  Navy  Program  Executive  Office 
for  Integrated  Warfare  Systems,  which  attempted  to  provide  a 
technical  justification  to  show  the  test  program  did  not  require 
an  SDTS  to  adequately  assess  the  self-defense  capability  of  the 
DDG  51  Flight  III  Class  Destroyers.  DOT&E  found  that  the 
study  presented  a  number  of  flawed  justifications  and  failed  to 
make  a  cogent  argument  for  why  an  SDTS  is  not  needed  for 
operational  testing. 

DOT&E  recommends  equipping  an  SDTS  with  capabilities 
to  support  Aegis  Combat  System,  AMDR,  and  ESSM  Block  2 
OT&E  to  test  ship  self-defense  systems’  performance  in  the  final 
seconds  of  the  close-in  battle  and  to  acquire  sufficient  data  to 
accredit  ship  self-defense  performance  M&S.  The  estimated  cost 
for  development  and  acquisition  of  these  capabilities  over  the 
Future  Years  Defense  Program  is  approximately  $284  Million. 

Of  that,  $228  Million  would  be  recouped  after  the  test  program 
completes  by  installing  the  hardware  in  a  future  DDG  51 
Flight  III  Destroyer  hull.  The  Navy  previously  agreed  with  this 
“re-use”  approach  in  their  December  2005  Air  Warfare/Ship 
Self  Defense  Test  and  Evaluation  Strategy  stating  that  “. .  .upon 
completion  of  testing  and  when  compatible  with  future  test 
events,  refurbish  and  return  the  test  units  to  operational  condition 
for  re-use.” 

Multi-Stage  Supersonic  Targets  (MSST) 

The  Navy  initiated  a  $120  Million  program  in  2010  to  develop 
an  adequate  multi-stage  supersonic  target  (MSST)  required  for 
adequate  operational  testing  of  Navy  surface  ship  air  defense 
systems.  The  MSST  is  critical  to  the  DDG  1000  Destroyer, 
CVN-78  Aircraft  Carrier,  DDG  51  Flight  III  Destroyer,  AMDR, 
Ship  Self-Defense  System,  Rolling  Airframe  Missile  Block  2,  and 
ESSM  Block  2  operational  test  programs.  The  MSST  program  is 
currently  undergoing  a  re-structure/baseline  to  address  technical 
deficiencies  as  well  as  cost  and  schedule  breaches  that  will 
postpone  its  initial  operational  capability  (IOC)  to  late  CY19. 
FY15  decrements  to  the  MSST  program  will  delay  MSST  IOC 
until  late  CY20,  further  delaying  the  completion  of  operational 
testing  for  those  programs. 

Fifth-Generation  Aerial  Target 

Current  aerial  targets,  including  the  QF-16  (in  development)  and 
sub-scale  drones,  do  not  adequately  represent  fifth-generation 
fighter  capabilities,  including  low  observability,  low  probability 
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of  intercept  sensors,  and  embedded  electronic  attack.  Aerial 
targets  with  the  capacity  to  represent  these  characteristics  are 
required  for  the  operational  test  adequacy  of  U.S.  air-to-air 
and  surface-to-air  weapons  systems.  With  the  projected 
deployment  of  foreign  fifth-generation  fighters  in  the  next 
five  years,  the  feasibility  of  completing  operationally  realistic 
testing  will  decline  significantly  without  a  new  full-scale 
aerial  target  solution.  The  risk  to  the  DOD  in  assessing  the 
mission  effectiveness  of  surface-to-air  and  air-to-air  missile 
weapon  systems  will  be  unacceptable  without  a  representative 
fifth-generation  aerial  target.  Over  the  next  decade,  the 
production  and  proliferation  of  foreign  fifth-generation 
fighter  aircraft  will  enhance  their  Anti-Access/Area  Denial 
capabilities  and,  without  question,  challenge  U.S.  air  superiority 
in  future  conflicts.  Current  weapon  system  testing  is  limited 
to  segmented  approaches  using  a  combination  of  captive 
carry  against  the  F-22  and  live -fire  against  sub-scale  and 
fourth-generation  full-scale  aerial  targets.  The  capacity  to 
conduct  end-to-end  testing,  from  weapon  system  radar  acquisition 
and  tracking,  missile  launch,  and  post-launch  seeker  acquisition 
to  end-game  fusing  against  a  fifth-generation  fighter  threat  with 
electronic  attack  capabilities  does  not  exist  and  constitutes  a 
critical  shortfall. 

DOT&E  initiated  studies  in  2006  on  the  design  and  fabrication 
of  a  dedicated  fifth-generation  aerial  target  to  evaluate  U.S. 
weapon  systems  effectiveness.  The  study  team,  comprised  of 
Air  Force  and  Navy  experts,  retired  Skunk  Works  engineers, 
and  industry,  completed  a  preliminary  design  review  for  a 
government-owned  design.  DOT&E  requested  $83  Million  in 
the  FY15  program  review  to  complete  final  design,  tooling,  and 
prototyping  efforts.  U.S.  industry  and  the  Canadian  Government 
informally  expressed  interest  in  potential  public-private 
partnership  opportunities  to  develop  this  target  system. 

Torpedo  Surrogates  for  Operational  Testing  of  Anti-Submarine 
Warfare  (ASW)  Platforms  and  Systems 

Operational  testing  of  ASW  platforms  and  related  systems 
includes  the  ability  to  detect,  evade,  counter,  and/or  destroy 
an  incoming  threat  torpedo.  The  determination  of  system  or 
platform  performance  is  critically  dependent  on  a  combination 
of  the  characteristics  of  the  incoming  torpedo  (e.g.,  dynamics, 
noise,  fusing,  sensors,  logic,  etc.).  Due  to  differences  in 
technological  approach  and  development,  U.S.  torpedoes  are 
not  representative  in  many  of  these  torpedo  characteristics  for 
many  highly  proliferated  torpedoes,  particularly  those  employed 
in  Anti-Surface  Warfare  by  other  nations.  Operational  testing 
that  is  limited  to  U.S.  exercise  torpedoes  will  not  allow  the 
identification  of  existing  limitations  of  ASW  systems  and  related 
systems  against  threat  torpedoes  and  will  result  in  uninformed 
decisions  in  the  employment  of  these  same  systems  in  wartime. 

A  January  9,  2013  DOT&E  memorandum  to  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  &  Acquisition) 
identifies  specific  threat  torpedo  attributes  that  the  threat  torpedo 
surrogate(s)  must  be  evaluated  against.  The  non-availability  of 
threat-representative  torpedo  surrogates  will  prevent  adequate 


operational  testing  for  ASW  platforms  and  related  systems,  as 
well  as  adversely  affect  tactics  development  and  validation  of 
these  tactics  within  the  fleet. 

Naval  Undersea  Warfare  Center  (NUWC)  Division  Keyport 
commenced  a  study  of  threat  torpedo  surrogates  in  FY14.  The 
$480,000  study  is  jointly  funded  by  the  Navy  and  DOT&E.  The 
study  focuses  on  the  identification  of  capability  gaps  between 
existing  torpedo  surrogates  and  threat  torpedoes.  The  study  will 
provide  an  analysis  of  alternatives  for  improvements  to  current 
torpedo  surrogates  and  development  of  new  torpedo  surrogates 
that  address  critical  gaps  in  threat  representation. 

NUWC  Division  Keyport  is  also  pursuing  a  prototype  technology 
development  project  that  will  deliver  a  threat-representative 
high-speed  quiet  propulsion  system.  The  development  of  a 
propulsion  system  prototype  will  leverage  the  critical  gaps 
identified  in  the  torpedo  threat  surrogate  capability  gap  analysis, 
discussed  in  the  preceding  paragraph.  This  effort  is  funded  by 
DOT&E  at  approximately  $1.0  Million  with  delivery  in  4QFY16. 
The  NUWC  Division  Keyport  study  and  prototype  development 
could  support  future  development  of  a  threat  torpedo  surrogate. 
Procurement  of  adequate  threat  torpedo  surrogates,  however,  is 
dependent  on  future  Navy  decisions.  DOT&E  believes  further 
development  and  production  of  threat  torpedo  surrogates  will 
benefit  from  an  enterprise  approach  to  prevent  burdening  a  single 
acquisition  program. 

Submarine  Surrogates  for  Operational  Testing  of  Lightweight 
and  Heavyweight  Torpedoes 

The  Navy  routinely  conducts  in-water  operational  testing  of 
lightweight  and  heavyweight  ASW  torpedoes  against  manned 
U.S.  Navy  submarines.  Although  these  exercise  torpedoes  do 
not  contain  explosive  warheads,  peacetime  safety  rules  require 
that  the  weapons  run  above  or  below  the  target  submarine  with 
a  significant  depth  stratum  offset  to  avoid  collision.  While  this 
procedure  allows  the  torpedo  to  detect,  verify,  and  initiate  homing 
on  the  target,  it  does  not  support  assessment  of  the  complete 
homing  and  intercept  sequence.  One  additional  limitation  is  the 
fact  that  U.S.  nuclear  attack  submarines  may  not  appropriately 
emulate  the  active  target  strength  (sonar  cross-section)  of  smaller 
threats  of  interest,  such  as  diesel-electric  submarines.  During  the 
Mk  50  lightweight  torpedo  operational  test,  the  Navy  conducted 
some  limited  set-to-hit  testing  against  manned  submarines,  which 
included  impact  against  the  target  hull,  but  that  practice  has  been 
discontinued. 

In  preparation  for  the  2004  Mk  54  lightweight  torpedo 
operational  test,  DOT&E  supported  the  development  and 
construction  of  the  unmanned  Weapon  Set-to-Hit  Torpedo 
Threat  Target  (WSTTT)  using  Resource  Enhancement  Project 
funding.  The  WSTTT  was  a  full-sized  steel  mockup  of  a  small 
diesel-electric  submarine,  with  an  approximate  program  cost  of 
$11  Million.  As  a  moored  stationary  target,  the  WSTTT  was 
limited  in  its  ability  to  emulate  an  evading  threat  but  its  use  in 
the  Mk  54  operational  test  demonstrated  the  value  of  such  a 
dedicated  resource.  Unfortunately,  the  Navy  did  not  properly 
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maintain  the  WSTTT  and  abandoned  it  on  the  bottom  of  the  sea 
off  the  California  coast  in  2006.  In  subsequent  years,  the  Navy 
was  able  to  make  some  limited  use  of  the  WSTTT  hulk  as  a 
bottomed  target  for  torpedo  testing. 

In  a  separate  effort,  the  Navy  built  the  Mobile  Anti-Submarine 
Training  Target  (MASTT),  designed  to  serve  as  a  full-sized  threat 
surrogate  for  use  in  training  by  surface  and  air  ASW  forces.  The 
Chief  of  Naval  Operations  initiated  the  program  in  2010  with  the 
goal  of  achieving  operational  capability  by  late  2011.  After  four 
years  and  an  expenditure  of  approximately  $15  Million,  the  Navy 
has  yet  to  use  the  MASTT  in  training  and  seems  to  be  on  the 
brink  of  abandoning  the  asset.  The  Navy  resisted  design  input 
from  the  operational  test  community  and  made  it  clear  that  the 
MASTT  was  not  intended  to  support  torpedo  testing. 

In  support  of  a  2010  Urgent  Operational  Need  Statement, 
the  Navy  funded  the  construction  of  the  Steel  Diesel-Electric 
Submarine  (SSSK),  a  full-sized  moored  set-to-hit  target 
consisting  of  an  open  steel  framework  with  a  series  of  corner 
reflectors  to  provide  appropriate  sonar  highlights.  The  Navy 
used  the  SSSK  as  a  target  for  the  Mk  54  torpedo  in  a  2011 
Quick  Reaction  Assessment  and  2013  FOT&E.  As  part  of  the 
Test  and  Evaluation  Master  Plan  approval  for  the  latter,  DOT&E 
sent  a  memorandum  indicating  that  the  Navy  must  develop  an 
appropriate  mobile  target  to  support  future  Mk  54  testing. 

Since  early  2013,  DOT&E  has  participated  in  a  Navy  working 
group  attempting  to  define  the  requirements  for  a  mobile 
set-to-hit  torpedo  target.  The  group  has  identified  a  spectrum  of 
options  and  capabilities,  ranging  from  a  torpedo-sized  vehicle 
towing  a  long  acoustic  array  to  a  full-sized  submarine  surrogate. 
At  the  very  least,  the  target  is  expected  to  be  mobile,  autonomous, 
and  certified  for  lightweight  torpedo  set-to-hit  scenarios.  More 
advanced  goals  might  include  realistic  active  and  passive  sonar 
signatures  to  support  ASW  search  and  reactive  capability  to 
present  a  more  realistically  evasive  target.  Cost  estimates  range 
from  under  $10  Million  for  a  towed  target  to  over  $30  Million  for 
a  full-sized  submarine  simulator. 

Threat  Modeling  and  Simulation  (M&S)  to  Support  Aircraft 
Survivability  Equipment  (ASE)  Testing 

Building  actual  threat  representations  for  widespread  testing  is 
expensive;  therefore,  DOT&E  focused  on  funding  incremental 
efforts  that  advance  the  use  of  authoritative  threat  M&S 
for  systems  T&E.  Although  threat  M&S  capabilities  have 
been  used  in  T&E  for  many  years,  these  were  not  always 
threat-representative,  and  different  M&S  instantiations  of  the 
same  threats  produced  different  results.  DOT&E’s  objective  is  to 
improve  the  consistency  of  threat  M&S  at  various  T&E  locations 
while  reducing  overall  costs. 

Throughout  the  T&E  processes,  M&S  represents  threats 
when  actual  threat  components  are  not  available;  provides 
more  complete  testing  than  possible  through  open-air  and 
hardware-in-the-loop  test  facilities;  and  provides  testing  when 
operational  reasons  such  as  flight  safety  preclude  physical  tests, 
especially  with  crew.  For  example,  test  programs  may  conduct 


10  -  20  live  threat  missile  firings  using  actual  threats.  Using 
threat  M&S  extends  those  results  across  a  much  larger  range, 
typically  20,000  cases  covering  different  threats,  ranges,  altitudes, 
aspect  angles,  atmospheric  conditions,  and  other  environmental 
variables  affecting  weapon  system  performance. 

DOT&E  implemented  controls  and  distribution  management  for 
threat  M&S  to  ensure  integrity  for  realistic  T&E  and  to  ensure 
test  results  were  not  affected  by  using  various  threat  M&S 
across  T&E  regimes.  The  T&E  Threat  M&S  Configuration 
Management  System  provides  mechanisms  to  effectively  identify 
and  correct  anomalies  between  threats  and  threat  representations, 
maintain  critical  documentation  such  as  interface  descriptions 
and  validation  documents,  control  model  configuration 
changes,  and  disperse  updated  threat  M&S  to  multiple  T&E 
facilities  for  consistency.  The  T&E  Threat  M&S  Configuration 
Control  Board,  comprised  of  representatives  from  intelligence 
organizations  and  the  T&E  community,  prioritizes  existing  threat 
M&S  developments  and  changes  to  ensure  updates  are  provided 
efficiently  to  T&E  user  facilities.  Requests  for  T&E  threat  M&S, 
report  anomalies,  or  request  changes  are  managed  through  an 
interface  on  DOD’s  Secret  Internet  Protocol  Router  Network. 

During  FY14,  the  T&E  Threat  Resource  Activity  provided 
standardized  and  authoritative  threat  M&S  to  multiple  T&E 
facilities  operated  by  the  Army,  Navy,  and  Air  Force  who 
implemented  them  into  various  T&E  uses  supporting  Aircraft 
Survivability  Equipment  (ASE)  testing.  DOT&E  also  engaged 
close  U.S.  allies  to  implement  same  threat  M&S  for  allied  T&E, 
leveraging  worldwide  implementation  of  standard,  authoritative 
threat  M&S  capabilities  for  T&E. 

DOT&E  also  developed  a  threat  M&S  roadmap  for  ASE  T&E 
to  provide  a  comprehensive  plan  and  to  prepare  future  test 
capabilities  using  standardized  and  authoritative  threat  M&S. 

For  example.  Joint  Standards  Instrumentation  Suite  captures 
threat  data  from  live  fire  test  events  to  support  threat  M&S 
development.  Starting  with  a  systematic  analysis  of  problems 
and  projects  that  support  effective  testing,  the  roadmap  lays 
out  a  path  for  the  development  of  threat-representative  test 
M&S  to  support  U.S.  and  allied  missile  warning  and  infrared 
countermeasure  systems.  DOT&E  estimates  that  $10  to 
13  Million  will  be  needed  between  FY16  -  20  to  fully  implement 
this  roadmap. 

Foreign  Materiel  Acquisition  Support  for  T&E 

DOT&E  is  responsible  for  ensuring  U.S.  weapons  systems 
are  tested  in  realistic  threat  environments.  Ideally,  operational 
testing  should  use  actual  threat  systems  to  create  realistic  threat 
environments.  Because  limited  resources  are  available  to  acquire 
foreign  threats,  DOT&E  annually  develops  a  prioritized  list 
of  threat  requirements  tied  to  upcoming  testing  of  programs 
on  the  OSD  T&E  Oversight  List  and  submits  them  to  the 
Defense  Intelligence  Agency  Joint  Foreign  Materiel  Program 
Office.  These  requirements  are  consolidated  with  Service  needs 
and  then  processed  through  various  Service  and  intelligence 
community  collection  activities.  DOT&E  then  coordinates 
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with  the  Department  of  State  to  identify  sources  and  increase 
opportunities  to  acquire  foreign  materiel  for  use  in  operational 
test  and  evaluation. 

Foreign  materiel  requirements  span  all  warfare  areas,  but 
recently  DOT&E  has  placed  a  priority  on  the  acquisition  of 
Man-Portable  Air  Defense  Systems  (MANPADS)  to  address 
significant  threat  shortfalls  that  affect  testing  for  infrared 
countermeasures  (IRCM)  programs  like  Common  IRCM 
(CIRCM),  Large  Aircraft  IRCM  (LA1RCM),  and  Department 
of  the  Navy  (DoN)  LAIRCM.  In  many  cases,  hundreds  of 
MANPADS  are  required  for  the  development  of  threat  M&S, 
for  use  in  hardware-in-the-loop  laboratories,  and  for  live-fire 
T&E,  all  to  present  realistic  threats  to  IRCM  equipment.  Using 
actual  missiles  with  actual  missile  seekers  aids  evaluators 
in  determining  the  effectiveness  of  IRCM  equipment,  but  is 
also  invaluable  in  development  of  effective  countermeasures 
throughout  the  U.S.  weapon  system’s  life. 

Due  to  the  inherent  challenge  of  developing  reliable  sources  for 
foreign  materiel,  negotiating  the  acquisition  of  foreign  materiel, 
and  the  difficulty  of  using  annual  appropriations  for  foreign 
materiel  acquisitions,  DOT&E  recommends  establishment  of 
dedicated,  non-expiring  funding  authority  to  support  foreign 
materiel  acquisitions. 

Real  Time  Casualty  Assessment  (RTCA) 

Force-on-force  battles  between  tactical  units  are  the  best 
method  of  achieving  a  realistic  environment  in  which  to  conduct 
operational  testing  of  land  and  expeditionary  warfare  systems. 
Simulated  force-on-force  battles  must  contain  realism  to  cause 
Soldiers  in  their  respective  units  to  make  tactical  decisions 
and  react  to  the  real-time  conditions  on  the  battlefield.  Real 
Time  Casualty  Assessment  (RTCA)  systems  integrate  Live, 
Virtual,  and  Constructive  components  to  enable  these  simulated 
force-on-force  battles.  RTCA  capability  provides  a  means  for 
simulated  engagements  to  have  realistic  outcomes  based  on  the 
lethality  and  survivability  characteristics  of  both  the  systems 
under  test  and  the  opposing  threat  systems.  RTCA  systems  must 
replicate  the  critical  attributes  of  real-world  combat  environments 
such  as  direct  and  indirect  fires,  Improvised  Explosive  Devices 
and  mines,  realistic  battle  damage,  and  casualties.  RTCA 
systems  must  record  the  time-space  position  information  and 
firing,  damage,  and  casualty  data  for  all  players  in  the  test  event. 
Post-test  playback  of  these  data  provides  a  critical  evaluation 
tool  to  determine  the  combat  system’s  capability  to  support 
Soldiers/Marines  as  they  conduct  combat  missions. 

DOT&E  has  requested  that  Army  Test  and  Evaluation  Command 
(ATEC)  use  its  available  RTCA  capability  to  improve  operational 
realism  and  to  provide  RTCA  data  collection  and  post-event 
playback  in  support  of  the  operational  testing  of  land  combat 
systems.  During  FY14,  two  separate  systems.  The  ATEC  Player 
and  Event  Tracking  System  (TAPETS)  and  the  Homestation 
Instrumentation  Training  System  (HITS),  were  used  by  ATEC  to 
provide  RTCA.  Both  interface  with  the  Instrumentable  Multiple 
Integrated  Laser  Engagement  System  (I-MILES)  for  direct  fire 
engagement  simulation. 


TAPETS/I-MILES,  the  legacy  RTCA  system  operated  by  Army 
Operational  Test  Command  (OTC),  was  used  successfully 
during  Network  Integration  Evaluation  (NIE)  14. 1  and 
NIE  14.2.  The  birds-eye-view  playback  of  data  collected  during 
NIE  14.1  proved  to  be  instrumental  during  the  evaluation  of 
the  AN/PRC- 11 7G  radio,  and  data  collected  during  NIE  14.2 
supported  the  evaluation  of  the  AN/PRC- 155  Manpack  radio. 
ATEC  should  continue  to  work  to  optimize  the  current 
TAPETS/I-MILES  system  and  look  for  ways  to  reduce  its 
operational  costs.  The  Army  should  update  the  probability  of  kill 
tables  that  are  the  foundation  of  I-MILES  engagements,  as  they 
have  not  kept  pace  with  the  fielding  of  new  vehicles  and  onboard 
communication  and  networking  equipment. 

The  Army  developed  HITS  to  provide  tactical  engagement 
simulation  for  units  during  force-on-force  training;  this  capability 
already  exists  on  a  number  of  Army  training  installations. 
H1TS/I-MILES  were  used  to  support  the  Joint  Light  Tactical 
Vehicle  Limited  User  Test  at  Fort  Stewart,  Georgia.  This  is 
the  first  attempted  use  of  HITS  to  support  OT&E,  and  early 
indications  are  that  HITS  has  some  shortfalls  as  a  test  tool.  Most 
significantly,  the  HITS  system  is  not  able  to  save  its  database 
to  support  post-test  analysis  and  playback,  which  reduces  its 
effectiveness  as  an  evaluation  tool.  The  Army  should  make  the 
necessary  modifications  to  the  HITS  software  immediately  so  that 
it  can  continue  to  be  used  to  support  testing. 

These  proposed  near-term  improvements  to  HITS  and  TAPETS 
will  give  ATEC  the  flexibility  to  select  the  most  capable  and 
cost-effective  RTCA  instrumentation  available  based  on  where 
operational  test  is  being  conducted.  In  support  of  future  test 
requirements,  the  Army  created  a  new  program  within  the 
Project  Manager  Instrumentation,  Targets,  and  Threat  Systems 
called  RTCA  Integrated  Test  Live,  Virtual,  and  Constructive 
Environment  (ITLE).  1TLE  will  provide  a  much-needed  stream 
of  funding  to  address  the  shortfalls  identified  in  the  recent  ATEC 
RTCA  study.  These  shortfalls  include  improving  the  ability 
to  seamlessly  simulate  indirect  fire  weapons,  IEDs/mines,  and 
air-to-ground/ground-to-air  combat.  DOT&E  is  encouraged 
by  the  increase  in  resources  the  Army  has  dedicated  to  RTCA 
development  and  use.  RTCA  is  essential  to  realistic  force- 
on-force  testing  of  current  and  future  land  and  expeditionary 
warfare  systems,  and  DOT&E  requires  RTCA  for  systems  such 
as  Family  of  Light  Amphibious  Vehicles,  Bradley  and  Abrams 
Upgrades,  Armored  Multi-purpose  Vehicle,  AH-64E  Block  HI, 
Joint  Light  Tactical  Vehicle,  and  Stryker  upgrades.  The  estimated 
cost  to  make  the  necessary  improve  to  the  ATEC  RTCA  systems 
is  $40  Million  over  the  next  five  years.  The  Army  has  made  a 
commitment  that  is  commensurate  with  this  need. 

Joint  Urban  Test  Capability  (JUTC) 

Operations  in  urban  environments  present  unique  challenges 
to  Service  members  and  their  equipment.  Degraded  mobility, 
communications,  and  situational  awareness;  a  large  civilian 
presence;  the  risk  of  collateral  damage;  reduced  stand-off 
distances;  and  unique  threat  profiles  are  some  of  the  conditions 
present  during  urban  operations.  These  challenges  underpin 
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the  requirement  that  systems  be  tested  in  operationally  realistic 
urban  environments. 

The  Army  is  currently  developing  the  Joint  Urban  Test 
Capability  (JUTC)  at  White  Sands  Missile  Range  with  funding 
provided  by  the  OSD  Central  Test  and  Evaluation  Investment 
Program.  DOT&E  is  supportive  of  the  JUTC  requirement, 
but  the  proposed  physical  surface  urban  area  of  200  meters  by 
240  meters  is  not  large  enough  to  support  operational  testing 
of  mechanized  units  of  company  size  and  greater.  The  remote 
location  chosen  for  JUTC  will  make  support  of  operational 
testing  difficult,  which  could  limit  its  utilization.  DOT&E 
recommends  the  urban  area  be  expanded  to  the  JUTC  objective 
requirement  of  900  meters  by  900  meters  originally  proposed 
in  the  Urban  Environment  Test  Capability  study,  and  that  the 
proposed  location  be  reconsidered  to  support  future  operational 
test  events.  The  cost  of  the  current  JUTC  effort  is  estimated  at 
$75-95  Million. 

Hypersonic  Weapons  Test  Infrastructure 

After  60  years  of  research,  the  U.S.  is  on  the  verge  of 
developing  operational  hypersonic  weapons.  The  United 
States  is  not  alone  in  its  pursuit  of  these  capabilities  and,  as 
recently  noted  by  the  Acting  Assistant  Secretary  of  Defense 
for  Research  and  Engineering,  “We. .  .do  not  want  to  be  the 
second  country  to  understand  how  to  control  hypersonics.” 
Hypersonic  weapons  will  present  a  challenge  to  potential 
adversaries  that  have  invested  in  anti-access  and  area-denial 
capabilities.  Consequently,  the  U.S.  Air  Force  Chief  of  Staff  has 
identified  hypersonic  weapons  as  one  of  five  “game-changing” 
technologies  in  Air  Force  strategic  planning. 

The  current  U.S.  hypersonic  T&E  infrastructure  is  not  adequate 
to  accomplish  critical  operational  or  developmental  test 
objectives,  reduce  risk,  and  adequately  inform  acquisition 
decisions  for  hypersonic  weapon  programs.  There  are  gaps 
in  important  ground  test  capabilities  for  aero-propulsion, 
aerodynamic,  aerothermodynamic,  and  material  evaluation, 
and  in  test  assets  for  lethality,  sensor  integration  and  guidance, 
navigation,  and  control.  Current  flight  test  ranges  cannot 
support  over-the-horizon  testing  of  long-range  hypersonic 
weapons.  Modeling  &  simulation  tools  are  not  mature  enough  to 
supplement  ground  and  flight  testing. 

In  the  past  20  years,  over  half  of  the  nation’s  hypersonic 
T&E  ground  facilities  built  in  the  1950s  through  the  1970s  to 
support  U.S.  space  and  missile  programs  have  been  closed  or 
demolished.  Many  of  the  remaining  19  “critical,”  one  of  a  kind 
hypersonic  Research,  Development,  and  T&E  facilities  are  in 
poor  or  dilapidated  condition  from  fiscal  neglect.  For  example, 
in  one  of  the  most  critical  hypersonic  test  facilities,  plastic  tarps 
are  being  used  to  prevent  sensitive  equipment  from  damage 
by  rainwater  leaking  through  the  roof.  Adequate  hypersonic 
test  infrastructure  is  required  to  support  the  development  of 
engineers  and  technicians  skilled  in  hypersonics.  The  shrinking 
and  aging  workforce  is  currently  insufficient  to  support  future 
hypersonic  testing  needs. 


Without  additional  investment  in  hypersonic  T&E  infrastructure 
and  personnel,  hypersonic  weapon  acquisition  programs  will 
need  to  rely  on  expensive  and  high-risk  flight  tests,  without 
adequate  precursor  ground  testing.  Premature,  catastrophic 
termination  of  four  out  of  six  recent  test  flights  for  the  X-51, 
Advanced  Hypersonic  Weapon  and  Hypersonic  Technology 
Vehicle  2  underscore  that  cost  and  risk.  Existing  ground-based 
hypersonic  T&E  facilities  that  help  prevent  future  flight  test 
failures  are  already  overtaxed.  As  hypersonic  programs  mature, 
ground  test  requirements  will  increase.  The  Test  Resource 
Management  Center  (TRMC)  estimates  the  requirement  for 
additional  hypersonic  T&E  resource  investments  from  FY 16-20 
at  $330  Million. 

DOT&E  recommends  funding  these  investments  to  address 
existing  hypersonic  T&E  gaps,  and  to  better  maintain  current 
hypersonic  T&E  infrastructure,  without  which  the  U.S.  will 
risk  ceding  the  advantage  of  hypersonic  weapons  to  potential 
adversaries. 

Range  Sustainability 

DOT&E  must  advocate  for  the  testing  of  new  and  upgraded 
military  capabilities  in  the  most  realistic  threat-representative 
environment  possible.  Due  to  safety  and  security  imperatives, 
these  environments  are  limited  to  geographic  areas  set  aside 
for  military  testing  and  training.  DOD  test  and  training  ranges 
are  located  in  once-remote  and  relatively-undisturbed  areas 
of  the  country — the  same  areas  that  today  are  sought  after  for 
development  of  renewable  energy  and  associated  electrical  power 
transmission  infrastructure.  Yet  energy  encroachment  is  not 
the  only  impact  to  a  robust  and  sustainable  range  infrastructure. 
Other  factors  continue  to  challenge  DOD’s  ability  to  test 
advanced  weapons  systems  in  real-world,  open-air  environments 
throughout  systems’  operational  envelopes.  These  include 
populations  moving  into  these  same  areas,  incompatibility  issues 
from  urban  growth,  competition  for  resource  use  (e.g.,  water, 
land,  airspace,  frequency  spectrum),  an  increasing  number  of 
(and  associated  requirements  to  protect)  listed  and  candidate- 
threatened  and  endangered  species,  and  increased  government 
regulation.  Already,  test  envelopes  to  evaluate  weapons 
systems  are  constricted  due  to  increased  combat  radii  for  threat 
engagement.  The  Major  Test  and  Range  Facility  Base  is  also 
threatened  by  the  impacts  of  extreme  weather  and  potential  water 
shortages  and  other  effects  from  a  changing  climate. 

As  reflected  in  past  annual  reports,  DOT&E  has  engaged  on 
behalf  of  the  DOD  test  community  to  ensure  required  capabilities 
remain  available  to  test  DOD  systems’  effectiveness,  reliability, 
and  lethality.  Two  current  major  areas  of  concern  are  availability 
of  frequency  spectrum  (addressed  later  in  this  section)  and 
encroachment  posed  by  the  development  of  renewable  energy  and 
transmission  line  projects.  This  new  form  of  encroachment  risks 
the  Department’s  ability  to  test  systems  under  realistic  conditions 
using  operational  resources.  While  M&S  is  used  increasingly  for 
testing,  development  of  realistic  models  requires  data  that  can 
only  be  obtained  from  live  testing. 
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DOT&E  is  a  co-chair  of  the  DOD  Siting  Clearinghouse,  along 
with  the  Deputy  Under  Secretary  of  Defense  for  Installations 
and  Environment  and  the  Deputy  Assistant  Secretary  of  Defense 
for  Readiness  (DASD(Readiness)).  The  number  of  projects 
received  by  the  Clearinghouse  under  the  Federal  Aviation 
Agency  Obstruction  Evaluation  Airport  Airspace  Analysis 
(FAA/OE-AAA)  process  increased  by  13  percent  from  FY13  to 
FY14  (from  an  average  of  173/month  to  220/month).  Of  the  total 
number  of  projects,  14  currently  under  review  have  a  significant 
potential  impact  on  DOD  range  capabilities  in  the  absence  of 
acceptable  mitigations.  In  addition,  there  are  other  renewable 
energy  and  electrical  power  transmission  infrastructure  project 
that  have  potential  impact  to  test  capabilities  that  have  been 
addressed  with  other  Federal  Departments  and  Agencies. 

Mitigation  measures  such  as  curtailment  of  wind  turbine 
operations  during  test  periods,  identification  of  alternative 
siting  for  renewable  energy  infrastructure,  alternative  siting  for 
affected  tests,  and  updates  to  DOD  test  hardware  and  software 
are  considered  during  review  of  each  proposed  projects.  In 
addition,  research  is  being  pursued  to  determine  the  effect  various 
renewable  energy  projects  may  have  on  DOD’s  instrumentation 
capabilities,  such  as  from  electromagnetic  interference  from 
electrical  power  transmission  infrastructure,  and  glint  and  glare 
from  utility-scale  solar  energy  projects.  The  Department  has 
invested  significant  time  and  resources  over  the  past  three  years 
to  identify  the  impact  of  wind  turbines  on  ground-based 
and  airborne  radars,  and  this  investment  may  help  mitigate 
interference  of  wind  turbines  with  test  range  infrastructure. 
Additionally,  the  advent  of  electrical  power  transmission 
infrastructure  near  the  test  and  training  ranges  can  be  an 
obstruction  to  low-level  flight  tests,  and  mitigation  options  such 
as  burial  of  the  power  line  may  be  required. 

Over  the  coming  year,  DOT&E  will  continue  to  work  with 
the  Clearinghouse,  the  TRMC,  the  Deputy  Under  Secretary  of 
Defense  for  Installations  and  Environment,  DASD(Readiness), 
Military  Departments,  and  other  Federal  Departments  and 
Agencies,  such  as  the  FAA  and  Department  of  the  Interior,  to 
refine  processes  for  resolving  compatibility  issues  between 
renewable  energy  projects  and  DOD  test  and  training 
requirements. 

Continuing  Radio  Frequency  Spectrum  Concerns 

Test  range  use  of  frequency  spectrum  continues  to  be 
challenged  by  pressures  to  repurpose  spectrum  to  broadband 
wireless  and  other  uses  such  as  medical  telemetry  and  wireless 
microphone  use.  DOT&E  documented  the  pending  loss 
of  1755  -  1780  MegaHertz  (MFlz)  and  compression  into 
1780  -  1850  MFlz  in  its  FY13  Annual  Report.  Table  2  illustrates 
the  frequency  bands  used  for  test  and  evaluation  and  identifies 
resource  issues  and  their  potential  mitigations.  An  additional 
development  during  2014  is  the  DOD’s  work  to  implement 
its  Electromagnetic  Spectrum  Strategy  (EMS),  as  well  as 
understanding  of  how  implementation  will  affect  DOD  testing. 
Adequate  frequency  spectrum  is  a  critical  resource  for  testing. 


It  is  required  to  both  upload  and  download  test  data  between 
the  article  being  tested  to  test  instrumentation,  and  to  control 
resources  during  test  operations. 

The  spectrum  allocated  is  used  full  time  during  the  range  day 
(i.e.,  from  6:00am  to  6:00pm),  and  continued  unimpeded  use  is 
vital  to  accommodate  the  increasing  volume  of  test  data  (e.g.,  that 
of  the  F-35  JSF).  As  Table  2  points  out,  both  the  range’s  primary 
L-  and  S-bands  are  now  being  targeted  for  repurposing  to 
broadband  wireless  use.  The  cost  impacts  to  the  Services’  T&E 
infrastructure  for  transitioning  capabilities  from  the  L-band  are: 

•  Army  -  $27.7  Million  is  required  to  retrofit  the  Aerial 
Telemetry  Systems  (AMTs)  at  White  Sands  Missile  Range 
that  are  operating  in  the  1755  -  1850  MFlz  band.  With  the 
loss  of  the  lower  25  MFlz  (1755  -  1780  band),  the  proposed 
solution  is  to  compress  operations  of  the  AMTs  into  the  retained 
1780  -  1850  MFlz  band  without  having  to  relocate/transition 
into  another  spectrum.  $  1  Million  is  required  to  replace  three 
point-to-point  datalinks  at  Aberdeen  Test  Center  that  are 
operating  in  the  1755  -  1780  MFlz  band.  New  equipment  will 
be  installed  to  operate  in  the  4  GigaFIertz  (GHz)  (C-band)  to 
accommodate  testing  of  robotics  which  will  be  relocating  to 

4  GHz. 

•  Navy  -  $  1 80  Million  for  transitioning  Aeronautical  Mobile 
Telemetry  using  an  approved  transition  plan. 

•  Air  Force  -  $100  Million  over  5  to  8  years  to  modify  95 
antennas,  628  receivers,  and  53  transmitters  for  compressing 
aeronautical  telemetry  into  the  1780  -  1850  MHz  band. 

The  test  ranges’  primary  band  for  telemetry,  1435  -  1525  MHz, 
has  two  pressing  challenges.  The  first  is  from  pending  Federal 
Communication  Commission  rulemaking  to  allow  shared  use  with 
wireless  microphones  used  for  major  concerts  and  sports  events, 
and  the  second  from  proposed  World  Radiocommunications 
Conference  (WRC)  repurposing  for  worldwide  wireless  broadband 
use.  The  first  issue  can  be  mitigated,  as  has  been  worked 
between  DOD  and  industry,  through  adoption  of  use  agreements 
(such  as  not-to-interfere  agreements)  and  use  of  electronic  keys 
to  coordinate  use.  The  second  issue,  WRC  re-purposing  the 
spectrum  for  worldwide  broadband  use,  is  more  difficult  for 
the  test  ranges.  Canada  has  engaged  with  DOD  and  the  aircraft 
industry  to  define  protection  methodologies,  and  Mexico  has  been 
approached  to  work  mitigation  strategies.  Due  to  the  location  of 
many  of  the  test  ranges  in  the  Southwest  continental  United  States 
and  aircraft  manufacturers’  testing  proximate  to  the  U.S.  and 
Canadian  border,  repurposing  of  the  1435  -  1525  MHz  spectrum 
is  of  major  concern. 

The  second  most-used  band  for  test  range  telemetry  is  the 
2360  -  2390  MHz  spectrum.  The  issue  confronting  the  ranges 
is  the  assignment  of  adjacent  spectrum  2345  -  2360  MHz  for 
wireless  broadband  use.  This  problem  is  resolvable  if  the 
vendor  using  the  adjacent  spectrum  implements  International 
Telecommunications  Union  rules,  which  prescribe  out-of-band 
emissions  protection.  DOD  is  working  this  issue  with  both  the 
Federal  Communication  Commission  and  the  vendor. 
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Since  the  DOD  EMS  was  published  in  2013,  there  has  been 
ongoing  work  to  develop  implementation  action  plans.  Many 
of  these  action  plans  address  issues  the  test  community  has 
already  been  working,  such  as  securing  sufficient  frequency 
to  sustain  test  operations,  and  developing  technologies  to  use 
available  frequency  more  efficiently.  Apart  from  these  similar 
approaches,  other  action  plans  address  operational  use  of 
spectrum.  The  conceived  new  EMS  operational  environment  will 
influence  DOT&E  oversight  of  test  planning,  given  consolidation 
and  development  of  operational  spectrum  tools  (spectrum 
identification,  characterization,  assignment).  When  implemented, 
many  of  the  EMS  action  plans  could  “simplify”  the  operating 
environment  by  stipulating  clear  policy,  procedures,  and  master 
architectures,  and  eliminating  the  myriad  of  stove -piped  systems 
that  have  been  deployed.  Thus,  the  EMS  has  the  potential  to 


change  the  operating  environment  for  Spectrum  Dependent 
Systems,  and  the  ways  such  systems  are  operated  during  testing. 
The  TRMC,  acting  as  the  proxy  for  the  DOD  T&E  community, 
will  have  varying  degrees  of  participation  in  88  of  the  349  action 
plan  tasks  identified  by  DOD  (primary  responsibility  for  10,  and 
coordinating  responsibility  for  78). 

Frequency  spectrum  is  a  limited  resource  with  many  more 
demands  than  supply.  With  allocations,  both  domestically 
and  internationally,  being  repurposed  for  non-defense  wireless 
transmission  needs,  DOT&E  will  need  to  remain  actively 
engaged  with  DASD(DT&E),  TRMC,  and  the  DOD  Chief 
Information  Officer  to  ensure  frequency  spectrum  allocations  are 
sufficient  for  the  conduct  of  test  operations,  and  also  that  these 
operations  use  frequency  efficiently. 
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TABLE  2.  FREQUENCY  ALLOCATIONS  USED  FORTESTING  AND  DOD  RESOURCE  ISSUES  AND  POTENTIAL  MITIGATIONS 


Frequency 

Use 

Users 

Resource  Issue  and  Potential  Mitigation 

Notes 

406.1  -420  MHz 

Land  mobile  radio 

Test  control  and  field  ops 

1350-1390  MHz 

Time,  Space, 
Position 

Information 

Critical  to  almost  all  open-air  tests; 
range  surveillance  radar  (Air  Route 
Surveillance  Radar-4) 

1435-1525  MHz 

L-Band  Telemetry  - 
Primary  Telemetry 
Band 

SDB,  UH1  /AH,T-45,  SH-60,  VH-S,  V-22, 
F-18,  F-18E,  F-22,  F-35,  B-2,  F-16,  B-1, 
B-2,  B-52,  Global  Hawk 

•  Issue:  Wireless  microphone  use. 

•  Potential  Mitigation:  Alternate  user 
coordination  with  assigned  key  codes  for 
spectrum  access  in  allotted  time  periods. 

•  Issue:  WRC  assignment  to  worldwide 
wireless  broadband  use. 

•  Potential  Mitigation:  Ongoing 
negotiations  with  Canada  and  Mexico. 

The  Light  Squared  (satellite/terrestrial 
network  proposed  and  abandoned) 
proposal  targets  1 505  - 1 525  MHz 

1675 -1710  MHz 

Weather,  including 
wind  speed 
measurement 

Critical  to  almost  all  open-air  tests 

1755  to  1780 

L-Band  Telemetry 

F/EA-1 8G,  Aerostar,  ASVS,  SM-2,  RAM, 
SSRT,  Classified  UAV  (WSMR),  ARAV, 
X-47,  the  only  band  for  miss-distance 
indicators  used  to  score  missile  shots 

•  Issue:  Auction  pending 

•  Potential  Mitigation:  Use  relocation 
to  4400 -4940  MHz  and  5091  -5150 

MHz  with  Spectrum  Relocation  Fund 
reimbursement. 

1780-1 850  MHz 

L-Band  Telemetry 

F/EA-1 8G,  Aerostar,  ASVS,  SM-2,  RAM, 
SSRT,  Classified  UAV  (WSMR),  ARAV, 
X-47,  the  only  band  for  miss-distance 
indicators  used  to  score  missile  shots 

This  spectrum  will  be  auctioned  over 
the  next  1 0  years,  and  some  sharing 
has  been  proposed 

2200  -  2290  MHz 

S-Band  Telemetry 

AIM-9X,  AIM-1 20,  JAASM,  JDAM, 
WCMD,  JSOW,  SDB,  Aerostar,  ASVS, 
WSI,  6 DOF,  MDA,  Patriot,  SM-2, 
ATACMS,  F-1 5,  F-1 6,  F-22,  F-35,T-38, 

B-1 ,  B-2,  B-52,  C-1 7,  Global  Hawk, 

X-51  Waverider 

2360  -  2390  MHz 

Upper  S-Band 
Telemetry 

F-1 8 E/400,  E2-D,  P-8A,  Exdrone,  Silver 
Fox,  THAAD,  F-1 6,  F-22,  B-1 ,  B-2,  B-52, 
C-1 7,  Global  Hawk 

•  Issue:  AT&T  wireless  communications 
use  of  2345-2360  MHz  without  Out  of 

Band  Emissions  protections. 

•  Potential  Mitigation:  Pending 

2390-2395  MHz 

Upper  S-Band 
Telemetry 

F-18E/400,  E2-D,  P-8A,  Exdrone,  Silver 
Fox, THAAD,  F-16,  F-22,  B-1,  B-2,  B-52, 
C-1 7,  Global  Hawk 

Shared  for  additional  Upper  S-Band 
coverage 

2700  -  2900  MHz 

Range  surveillance 
radar 

Critical  to  almost  all  open-air  tests 

4400 -4940  MHz 

Range  Telemetry 

F-15SA,  F-1 5  (pending),  fixed 
point-to-point  microwave,  tactical 
radio,  UAV,  threat  simulators 

•  Issue:  Pending  Federal  Communications 
Commission(FCC)  Rulemaking. 

•  Potential  Mitigation:  FCC  has  allowed 
band  use 

Band  is  just  now  coming  into  use 

5091  -5150  MHz 
(Region  2:5091  - 
6700  MHz) 

Range  Telemetry 

F-15SA 

•  Issue:  Pending  FCC  Rulemaking. 

•  Potential  Mitigation:  FCC  has  allowed 
band  use 

Shared  with  Federal  Aviation 
Administration.  Band  is  just  now 
coming  into  use;  DoD  has  requested 
that  the  band  be  extended  to  5250 

MHz  when  1 755  - 1 850  MHz  is 
auctioned. 
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Joint  Test  and  Evaluation  (JT&E) 


The  primary  objective  of  the  Joint  Test  and  Evaluation  (JT&E) 
Program  is  to  provide  solutions  rapidly  to  operational  deficiencies 
identified  by  the  joint  military  community.  The  program  achieves 
this  objective  by  developing  new  tactics,  techniques,  and 
procedures  (TTPs)  and  rigorously  measuring  the  extent  to  which 
their  use  improves  operational  outcomes.  JT&E  projects  may 
develop  products  that  have  implications  beyond  TTPs.  Sponsoring 
organizations  submit  these  products  to  the  appropriate  Service  or 
Combatant  Command  as  doctrine  change  requests.  Products  from 
JT&E  projects  have  been  incorporated  into  joint  and  multi-Service 
documents  through  the  Joint  Requirements  Oversight  Council 
process  and  through  coordination  with  the  Air,  Land,  Sea 
Application  Center.  The  JT&E  Program  also  develops  operational 
testing  methods  that  have  joint  application.  The  program  is 
complementary  to,  but  not  part  of,  the  acquisition  process. 

The  JT&E  Program  has  two  test  methods  available  for  customers: 
the  traditional  Joint  Test  and  the  Quick  Reaction  Test  (QRT). 

The  traditional  Joint  Test  method  is,  on  average,  a  two-year  joint 
test  project,  preceded  by  a  six-month  Joint  Feasibility  Study.  A 
Joint  Test  involves  an  in-depth,  methodical  test  and  evaluation 
of  issues  and  seeks  to  identify  solutions.  DOT&E  funds  the 
sponsor-led  test  team,  which  provides  the  customer  periodic 
feedback  and  useful,  interim  test  products.  The  JT&E  Program 
charters  two  new  Joint  Tests  annually.  The  program  managed 
six  Joint  Tests  in  FY14  that  focused  on  the  needs  of  operational 
forces.  Projects  annotated  with  an  asterisk  (*)  completed  in  FY14. 

•  Joint  Advanced  Capability  Employment  (J-ACE)* 

•  Joint  Base  Architecture  for  Secure  Industrial  Control  Systems 
(J-BASICS) 

•  Joint  Counter  Low,  Slow,  Small  Unmanned  Aircraft  Systems 
(JCLU) 

•  Joint  Deployable  Integrated  Air  and  Missile  Defense 
(JDIAMD)* 

•  Joint  Tactical  Air  Picture  (JTAP) 

•  Unmanned  Aircraft  Systems  -  Airspace  Integration  (UAS-AI) 


QRTs  are  intended  to  be  less  than  a  year  in  duration  and  solve 

urgent  issues.  The  program  managed  19  QRTs  in  FY14: 

•  Command  and  Control  of  Ballistic  Missile  Defense 
(C2BMD) 

•  Cyber  Agility  and  Defensive  Maneuver  (CAADM) 

•  Electromagnetic  Battle  Management  Concept  of  Operations 
Development  and  Evaluation  (E-CODE)* 

•  En-Route  Mission  Command  Capability  (EMCC)* 

•  Heterogeneous  Sensor  Integration  (HSI)* 

•  Joint  Assessment  Doctrine  Evaluation  (JADE) 

•  Joint  Automated  Net-Centric  Satellite  Communications 
(SATCOM)  Electromagnetic  Interference  (EMI)  Resolution 
(J-ANSER) 

•  Joint  Cyberspace  Intelligence,  Surveillance,  and 
Reconnaissance  (JCISR) 

•  Joint  Decision  Support  -  Air  ( JDeS-A) 

•  Joint  Graphical  Rapid  Assessment  of  Mission  Impact 
(J-GRAMI)* 

•  Joint  Homeland  Mining  Prevention  and  Response  (JHMPR) 

•  Joint  Integration  of  Cyber  Effects  (J-ICE)* 

•  Joint  Integrated  Standoff  Weapons  Employment  (JISOWE) 

•  Joint  Logistics  Enterprise  Data  Sharing  (JLEDS)* 

•  Joint  National  Capital  Region  (NCR)  Air  Surveillance 
Concept  of  Operations  (CONOPS)  -  Accelerated  (JNASC-A) 

•  Joint  Positive  Hostile  Identification  (J-PHID)* 

•  Joint  Precise  Timing  (JPT) 

•  Joint  Sensor  Awareness  to  Target  Tracking  (J-SATT)* 

•  Mortuary  Affairs  Contaminated  Remains  Mitigation  Site 
(MACRMS) 

As  directed  by  DOT&E,  the  program  executes  special  projects 

that  address  DoD-wide  problems.  The  program  managed  one 

special  project  in  FY14: 

•  Joint  Personnel  Recovery  Collaboration  and  Planning 
(JPRCaP) 


JOINTTESTS 


JOINT  ADVANCED  CAPABILITY  EMPLOYMENT  (J-ACE)* 
(CLOSED  AUGUST  2014) 

Sponsor/Start  Date:  U.S.  Strategic  Command 
(USSTRATCOM)/ August  2011 

Purpose:  To  develop,  test,  and  evaluate  a  standardized  process  to 
support  the  Joint  Force  Commander’s  ability  to  employ  enhanced 
special  programs  to  overcome  complex  targeting  challenges. 

Products/Benefits: 

•  A  repeatable  operational  employment  process  that  will  enhance 
planning  by  developing,  evaluating,  and  coordinating  concepts 
of  employment  (CONEMP)  that  can  be  used  by  the  Joint 


Staff,  Combatant  Commands,  Services,  and  National  Security 
Agency  to  solve  complex  targeting  challenges 

•  Multiple  enhanced  special  program  CONEMPs  to  overcome 
complex  targeting  challenges  that  are  approved  and  signed  at 
the  General  Officer/Flag  Officer  level  and  maintained  by  the 
appropriate  Combatant  Command  or  Service  component 

•  Relevant  training  scenarios  and  vignettes 

•  Documented  effects  associated  with  techniques  against 
representative  targets 

•  Developed  CONEMPs  allow  for  expeditious  development 
of  concept  of  operations  (CONOPS)  and  improved  special 
program  capability  approval  packages 
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JOINT  -  BASE  ARCHITECTURE  FOR  SECURE  INDUSTRIAL 
CONTROL  SYSTEMS  (J-BASICS) 

Sponsor/Start  Date:  U.S.  Cyber  Command/February  2014 

Purpose:  To  develop,  test,  and  evaluate  Advanced  Cyber 
Industrial  Control  System  (ICS)  TTPs  to  improve  the  ability 
of  ICS  network  managers  to  detect,  mitigate,  and  recover  from 
nation-state-level  cyber  attacks. 

Products/Benefits: 

•  Resiliency  (fight-through  capability)  to  DOD  ICS  networks 
and  immediate  supporting  Information  Technology 
infrastructures 

•  Advanced  means,  in  the  form  of  TTPs,  for  ICS  network 
managers  to  detect  nation-state-level  presence  in  DOD  ICS 
networks;  mitigate  damage  to  underlying  processes  supported 
by  the  ICS  in  the  event  of  a  cyber  attack;  and  quickly  recover 
the  ICS  network  to  a  condition  free  of  any  nation-state-level 
cyber  presence 

•  Increased  Commander  confidence  resulting  from  the  ability 
of  ICS  managers  to  defend  ICS  networks  from  active 
nation-state-level  attacks,  ensuring  mission  readiness  of 
ICS-dependent  activities 

•  Policy  and  implementation  guidance  recommendations  on 
ICS  network  security  to  Commander,  U.S.  Cyber  Command 
and  USD(AT&L)/Installations  and  Environment  Business 
Enterprise  Integration 

JOINT  COUNTER  LOW,  SLOW,  SMALL  UNMANNED  AIRCRAFT 
SYSTEMS  (JCLU) 

Sponsor/Start  Date:  Air  Force/ August  2012 

Purpose:  To  develop,  test,  and  evaluate  integrated  air  and 
missile  defense  (1AMD)  operator  TTPs  that  increase  operators’ 
ability  to  detect,  track,  and  identify  adversary  low,  slow,  and 
small  Unmanned  Aircraft  Systems  (UAS)  and  provide  timely 
notification  to  the  Area  Air  Defense  Commander. 

Products/Benefits: 

•  TTPs  to  increase  the  operators’  ability  to  detect,  track,  and 
identify  this  UAS  threat  category 

•  Integration  of  information  from  National  Technical  Means  into 
a  tactical  datalink  to  support  situational  awareness  and  target 
identification 

•  Development  of  the  operational  architecture  and  organizational 
relationships  that  will  increase  the  sharing  of  tactical 
information  to  improve  the  operators’  ability  to  execute  the 
joint  engagement  sequence 

JOINT  DEPLOYABLE  INTEGRATED  AIR  AND  MISSILE  DEFENSE 
(JDIAMD)* 

(CLOSED  JUNE2014) 

Sponsor/Start  Date:  North  American  Aerospace  Defense 
(NORAD),  U.S.  Northern  Command  (USNORTHCOM),  Army 
Space  and  Missile  Defense  Command/August  2011 


Purpose:  To  develop  joint  planning  and  execution  processes  and 
procedures  for  deployable  IAMD  for  the  homeland. 

Produets/Benefits: 

•  IAMD  process  modeling  that  provides  a  comprehensive  view 
of  the  integrated  planning  and  execution  process 

•  NORAD  and  USNORTHCOM  current  operations  planning 
processes,  checklists,  and  procedures  for  IAMD 

•  Continental  NORAD  Region,  Alaska  NORAD  Region,  and  Air 
Forces  North  planning  and  execution  TTPs  for  IAMD 

•  Naval  Forces  North  and  Third  Fleet  planning  and  execution 
TTPs  for  naval  support  of  IAMD 

•  Army  North  planning  and  execution  TTPs  for  operational 
control  of  ground-based  IAMD  forces 

•  263rd  Army  Air  and  Missile  Defense  Command  planning  and 
execution  TTPs  for  IAMD 

JOINTTACTICAL  AIR  PICTURE  (JTAP) 

Sponsor/Start  Date:  U.S.  Pacific  Command 
(USPACOM)/F ebruary  2014 

Purpose:  To  develop,  evaluate,  and  validate  joint  TTPs  to 
improve  the  joint  air  picture  and  engagement  opportunities, 
which  decreases  the  risk  of  hostile  attack  and  fratricide. 

Products/Benefits: 

•  Link  1 6  implementation  procedures  that  reduce  radio 
frequency  network  loading  by  moving  participants  to  internet 
protocol  architecture 

-  Improves  combat  identification  consistency 

-  Increases  the  number  of  tracks  containing  fire  control 
quality  data 

-  Enhances  track  update  rates 

•  Multi-Service  TTPs  that  enhances  integrated  fire  control  for 
defensive  counter  air  engagements 

UNMANNED  AIRCRAFT  SYSTEMS  AIRSPACE  INTEGRATION 
(UAS-AI) 

Sponsor/Start  Date:  NORAD-USNORTHCOM,  and  the  Army 
Test  and  Evaluation  Command/August  2012 

Purpose:  Standardize  and  evaluate  procedures  to  effectively 
operate  UAS  in  the  National  Airspace  System  (NAS). 

Produets/Benefits: 

•  Standardized  procedures  for  predictably  operating  UAS  in  the 
NAS  under  routine,  lost  command  link,  lost  two-way  radio 
communications,  and  lost  sense  and  avoid  conditions 

•  A  common  lexicon  for  UAS  operations  in  the  NAS 

•  Partnership  and  collaboration  with  the  Federal  Aviation 
Administration  to  integrate  UAS  into  the  NAS  by  2015 
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QUICK  REACTION  TESTS 


COMMAND  AND  CONTROL  OF  BALLISTIC  MISSILE 
DEFENSE  (C2BMD) 

Sponsor/Start  Date:  Air  Force  Joint  Test  Office/February 
2014 

Purpose:  To  develop,  test,  and  refine  joint  TTPs  leveraging 
current  Command  and  Control,  Battle  Management  and 
Communications  (C2BMC)  capabilities  resident,  but  not 
fully  utilized,  to  enhance  intra-  and  inter-theater  joint  BMD 
operations  planning  and  re-planning  efforts. 

Products/Benefits : 

•  Improve  BMD  coordination  among  the  Air  Operations 
Center,  Maritime  Operations  Center,  and  Army  Air 
and  Missile  Defense  Command  in  support  of  intra-  and 
inter-theater  BMD  operations 

•  Enhance  the  ability  of  theaters  to  successfully  plan  and 
employ  limited  organic  BMD  assets 

•  Improve  use  of  capabilities  resident,  but  underutilized,  in 
fielded  C2BMC  6.4  software 

CYBER  AGILITY  AND  DEFENSIVE  MANEUVER  (CAADM) 

Sponsor/Start  Date:  USPACOM/August  2014 

Purpose:  To  develop,  test,  and  refine  TTPs  to  enhance 
moving  target  technologies  to  enable  cyber  agility  and 
defensive  cyber  maneuver  for  the  protection  of  selected 
critical  command  and  control  capabilities  and  information 
resources  from  advanced  threats.  Also,  to  provide 
recommendations  for  amendments  of  joint  doctrine 
(principally  Joint  Publication  3-12)  to  introduce  more 
comprehensive  operational  concepts  for  defensive  maneuver 
in  cyberspace. 

Products/Benefits : 

•  Develop  TTPs  and  recommend  changes  to  joint  doctrine  to 
provide  the  following: 

-  Assist  commanders  and  network  defenders  in 
overcoming  disadvantages  inherent  in  static  cyber 
defenses 

-  Decrease  vulnerability  to  enemy  surveillance  of  and 
attacks  against  DOD  network  enclaves 

-  Enhance  ability  to  rapidly  adapt  cyber  defenses  in  the 
face  of  changing  missions  and  threats 

-  Improve  capability  to  counter  and  observe  enemy 
actions  in  cyberspace 

-  Increase  wherewithal  to  shift  initiative  from  attackers  to 
network  defenders 

-  More  effective  application  of  technology  for  agile 
defense  of  key  terrain  in  cyberspace 

•  Will  underpin  more  effective  joint  planning  and 
operations  for  defense  of  critical  capabilities  and 
information  resources 


ELECTROMAGNETIC  SPECTRUM  BATTLE  MANAGEMENT 
CONCEPT  OF  OPERATIONS  DEVELOPMENT  AND 
EVALUATION  (E-CODE)* 

(CLOSED  MAY  2014) 

Sponsor/Start  Date:  USSTRATCOM/March  2013 

Purpose:  To  validate  a  CONOPS  for  establishing  a  Joint 
Electromagnetic  Spectrum  Operations  Cell  at  the  Combatant 
Command  or  Joint  Task  Force-level. 

Products/Benefits: 

•  Integrated  Joint  Electromagnetic  Spectrum  Operations  cell 
planning,  tasking,  coordination,  and  conflict  resolution 
processes 

•  Synchronized  operations  to  shape  the  electromagnetic 
battlespace  to  meet  the  commander’s  objectives 

•  Codified  processes  to  gain  and  maintain  freedom  of  movement 
in  the  electromagnetic  operating  environment  while  denying 
access  to  adversaries 

•  Improved  information  exchange,  situational  awareness,  and 
command  and  control  decision  processes  to  reduce  the  timeline 
for  dynamic  reallocation  of  the  congested  and  contested 
electromagnetic  spectrum 

•  Improved  processes  for  prioritizing,  nominating,  and 
neutralizing  electromagnetic  spectrum  targets 

EN-ROUTE  MISSION  COMMAND  CAPABILITY  (EMCC)* 
(CLOSED  JULY  2014) 

Sponsor/Start  Date:  XVIII  Airborne  Corps/May  2013 

Purpose:  To  develop,  test,  and  refine  TTPs  for  installation 
and  operational  use  of  a  robust  EMCC  that  provides  global 
response  forces  with  the  ability  to  establish  and  maintain  optimal 
situational  awareness  while  airborne,  en-route,  and  on  the  ground 
to  conduct  forcible  entry  operations. 

Products/Benefits: 

•  Formalized  TTPs  for  EMCC  installation  and  operation  for  the 
82nd  Airborne  Division  and  75th  Ranger  Regiment 

•  Provided  guidance  for  leveraging  EMCC  to  support  forcible 
entry  operations 

•  Developed  supporting  architectures  for  EMCC  connectivity 

HETEROGENEOUS  SENSOR  INTEGRATION  (HSI)* 

(CLOSED  JULY  2014) 

Sponsor/  Start  Date:  USPACOM/March  2013 

Purpose:  To  develop  and  test  TTPs  for  training,  alignment,  and 
integration  of  experimental  sensors  with  existing,  signature -based 
sensors  to  enhance  situational  awareness  of  key  terrain  in 
cyberspace.  The  objective  is  to  demonstrate,  through  the  test 
and  evaluation  process,  a  significant  reduction  in  false-positive 
reporting  and  an  improvement  in  the  precision  of  detection 
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of  intrusions  when  employing  heterogeneous  sensor  pairs  in 
accordance  with  the  HSI-developed  TTPs. 

Products/Benefits: 

•  Improved  network  defenders’  detection  rates  while  reducing 
false-positive  alert  rates  associated  with  network  intrusions, 
enhancing  the  Joint  Force  Commander’s  situational  awareness 
of  key  terrain  in  cyberspace 

•  The  associated  operational  CONEMP  describes  when  and 
where  it  is  appropriate  to  employ  the  HSI-developed  TTPs  by 
showing  how  the  capability  fits  within  the  broader  context  of 
joint  operations 

JOINT  ASSESSMENT  DOCTRINE  EVALUATION  (JADE) 

Sponsor/Start  Date:  U.S.  European  Command;  USPACOM; 

U.S.  Special  Operations  Command;  Center  for  Army  Analysis, 
Army  Materiel  System  Analysis  Activity/February  2014 

Purpose:  Refinement  and  validation  of  the  assessment  process 
and  methodology  developed  from  observations  at  the  joint  forces 
level,  and  identifying  how  to  better  integrate  assessments  into 
existing  plans  and  operations  processes.  The  process  will  initially 
look  at  using  a  checklist  guide  in  a  handbook  format  aimed 
at  the  end  user,  working  closely  with  the  Joint  Staff  J7  Joint 
Doctrine  Note  on  assessments  to  vet  and  synchronize  identified 
assessments  process  and  methodology. 

Products/Benefits:  Standardize  processes  and  doctrinal  guidance 
to  assist  planners  and  operators  on  how  to  develop  an  assessment 
strategy  for  identifying  indicators,  triggers  to  decision  points,  and 
methods  to  measure  progress  (or  lack  of  it)  towards  a  desired  end 
state. 

JOINT  AUTOMATED  NET-CENTRIC  SATCOM  EMI  RESOLUTION 
(J-ANSER) 

Sponsor/Start  Date:  USSTRATCOM,  Air  Force  Joint  Test 
Office/November  2013 

Purpose:  To  develop  SATCOM  EMI  TTPs  leveraging  recently 
fielded  net-centric  systems  to  immediately  detect,  characterize, 
and  cue  geolocation  assets. 

Produets/Benefits: 

•  Allows  commanders  and  operators  to  advance  operations  in 
a  SATCOM-denied  and/or  degraded  environment  by  visually 
displaying  SATCOM  lines  of  communication  health  and  status 

•  Improve  SATCOM  operators  responsiveness  to  SATCOM 
interference  or  breadth  of  attack 

•  Enhance  real-time  situational  awareness  by  reducing 
SATCOM  EMI  resolution  timelines  from  hours  to  minutes 

JOINT  CYBERSPACE  INTELLIGENCE,  SURVEILLANCE  AND 
RECONNAISSANCE  (JCISR) 

Sponsor/Start  Date:  USPACOM/January  2014 

Purpose:  To  develop  and  evaluate  TTPs  that  enable  a  joint  cyber 
center  to  integrate  the  Intelligence  Community  (IC)  cyberspace 
ISR  into  joint  operation  planning,  joint  targeting,  and  operations. 


Products/Benefits: 

•  Establish  and  refine  processes  for  full  integration  of  the  IC 
cyberspace  ISR  into  planning,  targeting,  and  execution  of 
offensive  cyber  operations  by  Combatant  Commander’s  Cyber 
Mission  Forces  (CMFs) 

•  Provide  a  framework  for  the  IC  to  communicate  with 
newly-formed  Combatant  Command  CMFs 

•  Develop  a  doctrine,  organization,  training,  materiel,  leadership 
and  education,  personnel,  facilities,  and  policy  change 
request  on  factors  that  impede  IC  support  for  offensive  cyber 
operations 

•  Validate  TTPs  through  an  assessment  of  developed  processes 
across  Combatant  Commands 

JOINT  DECISION  SUPPORT  -  AIR  (JDES-A) 

Sponsor/Start  Date:  NORAD/November  2013 

Purpose:  To  develop,  test,  and  refine  TTPs  for  use  by  operators 
of  the  Air/Event  Information  Sharing  Service  (A/EISS)  to  bring 
A/EISS  formally  into  the  Operation  NOBLE  EAGLE  threat 
decision  processes  and  to  achieve  its  full  operational  capability. 

Produets/Benefits: 

•  Enhanced  A/EISS  TTPs  to  deliver  air-domain  situational 
awareness  and  decision  support  and  enable  the  Commander, 
NORAD-USNORTHCOM,  Civil  Aircraft  Engagement 
Authorities,  Canadian  Recommending  Authorities,  and 

all  participating  joint,  interagency,  intergovernmental,  and 
multinational  air  defense  and  security  mission  partners  to 
make  responsible  and  timely  decisions  during  air  events  over 
North  America 

•  Training  and  evaluation  products  to  ensure  implementation  of 
TTPs  is  consistent  and  reliable 

JOINT  GRAPHICAL  RAPID  ASSESSMENT  OF  MISSION  IMPACT 
(J-GRAMI)* 

(CLOSED  MARCH  2014) 

Sponsor/Start  Date:  USSTRATCOM/December  2012 

Purpose:  To  develop  and  evaluate  TTPs  for  mission  impact 
documentation,  collaboration,  and  visualization  of  problem  sets 
for  USSTRATCOM’s  space  missions  and  nuclear  command  and 
control.  The  TTPs  leveraged  the  proof-of-concept  Graphical 
Mission  Impact  Tool  that  USSTRATCOM’s  Mission  Assurance 
Division  created  to  graphically  display  mission  effects 
resulting  from  loss  or  disruption  of  critical  systems,  assets,  and 
infrastructure. 

Produets/Benefits: 

•  TTPs  that  provide  USSTRATCOM  and  USPACOM  an 
operational  mission  impact  evaluation  methodology  for  loss  or 
disruption  of  critical  systems,  assets,  or  infrastructure 

•  Extensive  detailed  directions  for  using  the  Graphical  Mission 
Impact  Tool  to  do  the  following: 

-  Dynamically  identify  vulnerabilities  in  critical  systems, 
assets,  and  defense  infrastructure  needed  to  support 
assigned  missions  and  mission-essential  tasks 
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-  Assess  and  graphically  represent  potential  impacts 
resulting  from  loss  or  disruption  of  critical  systems,  assets, 
or  infrastructure 

-  Provide  decision  makers  with  rapid  capability  to  support 
informed  decision  making 

JOINT  HOMELAND  MINING  PREVENTION  AND  RESPONSE 
(JHMPR) 

Sponsor/Start  Date:  NORAD-USNORTHCOM/August  2013 

Purpose:  To  develop  a  standardized,  cross-departmental, 
interagency  CONOPS  to  support  the  expediency  and  accuracy 
of  the  information  necessary  to  better  enable  detection  and  mine 
countermeasures  (MCM)  response  to  a  mine  or  an  underwater 
improvised  explosive  device  (UWIED)  incident  in  the 
United  States. 

Products/Benefits: 

•  CONOPS  for  the  rapid,  accurate,  and  standardized  information 
exchange  between  the  Department  of  Homeland  Security 
mission  of  mine/UWIED  detection  and  prevention  and  the 
DOD’s  MCM  response 

•  Timely  and  effective  MCM  responses,  which  will  better  allow 
the  quick  restoration  of  commerce  and  port  security 

JOINT  INTEGRATION  OF  CYBER  EFFECTS  (J-ICE)* 

(CLOSED  NOVEMBER  201 3) 

Sponsor/Start  Date:  USPACOM/October  2012 

Purpose:  To  develop  and  evaluate  TTPs  that  enable  a  joint  cyber 
center  to  integrate  cyber  effects  into  joint  planning,  targeting,  and 
operations. 

Products/Benefits: 

•  Established  and  refined  processes  for  planning,  targeting,  and 
executing  offensive  cyber  operations 

•  Enabled  the  Combatant  Commander’s  application  of 
operational  art  to  project  cyber  power’s  capability  to  achieve 
an  objective 

•  Provided  a  framework  for  command  and  control  of 
newly-formed  cyber  forces  within  the  command 

•  Developed  inputs  for  doctrine,  organization,  training,  materiel, 
leadership  and  education,  personnel,  and  facilities  on  factors 
that  impede  planning  for  offensive  cyber  operations 

•  Validated  TTPs  through  an  assessment  of  developed  processes 
across  the  Combatant  Commands 

JOINT  INTEGRATED  STANDOFF  WEAPONS  EMPLOYMENT 
(JISOWE) 

Sponsor/Start  Date:  USPACOM/February  2014 

Purpose:  To  develop  and  evaluate  TTPs  enabling  more  efficient 
and  effective  employment  of  scarce  stand-off  resources  by 
enhancing  mission  planning  and  analysis  for  execution  with  the 
right  number  of  standoff  weapons  at  the  right  place  and  at  the 
right  time  to  ensure  successful  mission  accomplishment  and 
minimize  fratricide. 


Products/Benefits: 

•  Qualitative  and  empirical  data  analysis  to  support  findings, 
conclusions,  and  recommendations  for  Combatant  Command 
leadership,  the  United  States  Air  Force  Warfare  Center, 

and  the  561st  Joint  Tactics  Squadron  leadership  on  system 
employment 

•  Effective  joint  TTPs  for  the  integrated  planning  and 
employment  of  joint  packages  of  standoff  munitions,  decoys, 
and  jammers 

JOINT  LOGISTICS  ENTERPRISE  DATA  SHARING  (JLEDS)* 
(CLOSED  APRIL  2014) 

Sponsor/Start  Date:  Joint  Staff,  U.S.  Transportation 
Command/ January  2013 

Purpose:  To  develop  and  test  credentialed-access,  web-based 
enterprise  interfaces  to  multiple  sources  of  data  regarding 
redeployment  and  retrograde  of  materiel  from  the  U.S.  Central 
Command  theater.  The  interface  will  present  these  data  with 
aggregated  or  detailed  visualizations. 

Products/Benefits: 

•  User  manual  and  TTPs  that  provide  step-by-step  guidance 
regarding  the  use  of  the  Cargo  Movement  Report  widget.  It 
describes  the  system  and  environments  in  which  the  widget 
resides  and  describes  the  functionality  existing  within  the 
widget  to  access  logistics  data  relevant  to  Afghanistan 
retrograde  operations 

•  The  JLEDS-developed  CONOPS  is  a  Microsoft  Word-based 
user  guide  document,  which  can  be  delivered  to  customers 
in  either  electronic  format  or  hard  copy.  The  CONOPS 
describes  how  an  Authoritative  Data  Source  (ADS)  can  expose 
logistics  data,  or,  for  a  user  organization  to  establish  a  process 
to  access  data  from  an  ADS.  It  includes  use  case  examples, 
requirements  definitions,  and  best  practices  regarding  systems 
architectures. 

JOINT  NATIONAL  CAPITOL  REGION  AIR  SURVEILLANCE 
CONCEPT  OF  OPERATIONS  -  ADVANCED  (JNASC-A) 

Sponsor/Start  Date:  NORAD-USNORTHCOM/August  2013 

Purpose:  To  develop  the  procedures  to  enable  Joint  Land  Attack 
Cruise  Missile  Defense  Elevated  Netted  Sensor  System  (JLENS) 
to  detect,  track,  and  characterize  items  of  interest  within  the 
JLENS  field  of  view  supporting  the  air  surveillance  of  the  NCR 
and  surrounding  areas  to  positively  identify  contacts  and  increase 
the  time  available  to  take  action  to  counter  air  and  missile  threats. 

Products/Benefits: 

•  The  JLENS  TTPs  provide  Eastern  and  Western  Air  Defense 
Sector  operators  efficient  and  effective  detection  and 
identification  capabilities  to  improve  the  NORAD  homeland 
defense  mission  of  the  NCR 

•  Hastens  the  ability  of  commanders  to  utilize  the  JLENS  for 
homeland  defense  by  expediting  the  development  of  joint 
TTPs  6  to  12  months  earlier  than  previously  possible 
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JOINT  POSITIVE  HOSTILE  IDENTIFICATION  (J-PHID)* 

(CLOSED  JUNE2014) 

Sponsor/Start  Date:  NORAD-USNORTHCOM/March  2013 

Purpose:  To  develop  and  evaluate  TTPs  to  improve  IAMD 
decision-making  processes  that  will  enable  faster  and  more 
accurate  responses  in  an  increasingly  dynamic  air-  and 
missile-defense  environment.  The  goal  of  this  QRT  was  to 
decrease  the  time  required  to  positively  identify  a  contact  of 
interest  and  increase  the  time  available  to  take  action  to  counter 
air  and  missile  threats. 

Products/Benefits: 

•  IAMD  TTPs  to  more  efficiently  and  effectively  execute  the 
joint  engagement  sequence  in  defense  of  the  homeland 

•  J-PHID-developed  algorithm  assigns  a  confidence  level 
to  a  contact  of  interest,  resulting  in  an  improved  IAMD 
decision-making  process,  reduced  response  time,  and  increased 
accuracy  while  executing  the  joint  engagement  sequence 

JOINT  PRECISE  TIMING  (JPT) 

Sponsor/Start  Date:  Office  of  the  Chief  of  Naval 
Operations/ August  2013 

Purpose:  To  develop,  test,  and  refine  TTPs  to  provide 
overarching  guidance  and  best  practices  for  the  standardization 
and  operation  of  DOD  Precision  Time  and  Time  Interval  (PTTI) 
systems. 

Produets/Benefits: 

•  TTPs  that  provide  overarching  guidance  for  the  standardization 
and  operation  of  DOD  PTTI  distribution  systems 

•  Formalizes  and  documents  the  best  practices  that  provide 

a  methodical,  repeatable,  and  verifiable  set  of  guidelines  to 
improve  the  reliability,  redundancy,  and  assurance  of  DOD 
systems  and  provide  system  installers,  maintainers,  and 
operators  with  a  more  comprehensive  understanding  of  PTTI 
effects  for  DOD  systems 

•  Enables  the  joint  force  to  achieve  and  sustain  accurate, 
synchronized  time  of  day  and  frequency  worldwide  to  support 
joint  operations 


JOINT  SENSOR  AWARENESS  TO  TARGETTRACKING  (J-SATT)* 
(CLOSED  MAY  201 4) 

Sponsor/Start  Date:  USPACOM/March  2013 

Purpose:  To  develop  and  evaluate  TTPs  for  the  rapid  injection 
of  fused  track  data  derived  from  the  Dynamic  Time  Critical 
Warfighting  Capability  into  available  tactical  datalinks. 

Products/Benefits: 

•  Provided  methods  to  disseminate  unverified  intelligence  over 
tactical  networks  for  timely  situational  awareness  of  mobile 
threats  to  warfighters 

•  The  datalink  message  supplements  the  original  track 
information,  specifically  highlighting  the  type  of  threat,  when  it 
was  last  detected,  and  accuracy 

MORTUARY  AFFAIRS  CONTAMINATED  REMAINS  MITIGATION 
SITE  (MACRMS) 

Sponsor/Start  Date:  U.S.  Army  Quartermaster 
School/February  2014 

Purpose:  To  test  and  evaluate  the  MACRMS  TTPs  during  a 
simulated  chemical,  biological,  radiological,  or  nuclear  incident  in 
order  to  accomplish  the  following: 

•  Capture  lessons  learned  and  best  practices  related  to  the 
operation  of  a  MACRMS 

•  Refine  and  validate  the  TTPs  for  joint  personnel  conducting  the 
MACRMS  mission,  including  hazard  mitigation,  processing 
remains,  performing  identification  tasks,  and  preparing 
contaminated  human  remains  for  evacuation 

•  Assess  personnel  protective  equipment,  site  equipment,  and 
training  support  for  the  safe  execution  of  the  MACRMS 
mission 

Products/Benefits: 

•  Validated  TTPs  for  processing  contaminated  human  remains 
and  their  personal  effects  resulting  from  a  chemical,  biological, 
radiological,  or  nuclear  incident  overseas  or  in  the  United  States 

•  Update  to  Mortuary  Affairs  school  curriculum  for  training 

•  Update  to  applicable  Service  and  joint  doctrine 


SPECIAL  PROJECTS 


JOINT  PERSONNEL  RECOVERY  COLLABORATION  AND 
PLANNING  (JPRCAP) 

Sponsor/Start  Date:  Joint  Personnel  Recovery 
Agency/ January  2013 

Purpose:  To  develop,  test,  and  evaluate  procedures  that  will 
formalize  planning,  crisis  response,  and  information  sharing 
between  the  Combatant  Commands,  senior  defense  officials,  and 
State  Department  defense  attaches  prior  to  and  during  personnel 
recovery  responses  where  a  State  Department  Chief  of  Mission 
(generally  the  ambassador),  and  not  a  DOD  official,  is  the  lead 
U.S.  Government  authority  for  activity  in  a  country. 


Products/Benefits: 

•  Processes  and  products  that  provide  fonnal  personnel 
recovery  planning  and  training  protocols  that  inform  senior 
defense  officials  and  State  Department  defense  attaches  in 
United  States  embassies,  in  coordination  with  the  Combatant 
Commands,  on  how  to  build  and  implement  country-specific 
personnel  recovery  plans  tailored  for  the  Chief  of  Mission 
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Center  for  Countermeasures 


The  Center  for  Countermeasures  (the  Center)  is  a  joint  activity 
that  directs,  coordinates,  supports,  and  conducts  independent 
countermeasure/counter-countermeasure  (CM/CCM)  test  and 
evaluation  (T&E)  activities  of  U.S.  and  foreign  weapon  systems, 
subsystems,  sensors,  and  related  components.  The  Center 
accomplishes  this  work  in  support  of  DOT&E,  Deputy  Assistant 
Secretary  of  Defense  for  Developmental  Test  and  Evaluation 
DASD(DT&E),  weapon  system  developers,  and  the  Services. 

The  Center’s  testing  and  analyses  directly  support  evaluation 
of  the  operational  effectiveness  and  suitability  of  CM/CCM 
systems.  Specifically,  the  Center: 

•  Performs  early  assessments  of  CM  effectiveness  against  threat 
and  DOD  systems  and  subsystems. 

•  Determines  performance  and  limitations  of  missile-warning 
and  aircraft  survivability  equipment  (ASE)  used  on 
rotary-wing  and  fixed-wing  aircraft. 

•  Determines  effectiveness  of  precision  guided  weapon  (PGW) 
systems  and  subsystems  when  operating  in  an  environment 
degraded  by  CMs. 

•  Develops  and  evaluates  CM/CCM  techniques  and  devices. 

•  Develops  and  tests  new  CMs  in  operationally  realistic 
environments. 

•  Provides  analysis  and  recommendations  on  CM/CCM 
effectiveness  to  Service  Program  Offices,  DOT&E, 
DASD(DT&E),  and  the  Services. 

•  Supports  Service  member  exercises,  training,  and 
pre -deployment  activities. 

During  FY14,  the  Center  completed  over  36  T&E  activities. 

The  Center’s  support  of  these  activities  resulted  in  analysis 
and  reporting  on  more  than  30  DOD  electro-optical  systems  or 
subsystems  with  special  emphasis  on  rotary-wing  survivability. 


The  Center  participated  in  operational/developmental 
tests  for  rotary-  and  fixed-wing  ASE,  PGWs,  hostile  fire 
indicator  (HFI)  data  collection,  experimentation  tests,  and 
pre-deployment/ exercise  support  involving  the  use  of  CM/CCM. 
To  best  represent  the  level  of  effort  resourced  to  support  T&E, 
the  Center  tracks  funding  expended  in  each  test  area. 

•  Approximately  55  percent  of  the  Center’s  efforts  were 
spent  on  ASE  testing,  with  the  majority  of  these  efforts  in 
support  of  rotary-wing  aircraft. 

•  Approximately  21  percent  of  the  Center’s  efforts  were  spent 
on  PGW,  foreign  system,  and  other  types  of  field  testing  not 
related  to  ASE. 

•  Approximately  seven  percent  of  the  Center’s  efforts  were 
dedicated  to  overseas  contingency  operations  support, 
with  emphasis  on  CM-based  pre-deployment  training  for 
rotary- wing  units. 

•  Approximately  15  percent  of  the  Center’s  efforts  were  spent 
on  internal  programs  to  improve  test  capabilities  and  develop 
test  methodologies  for  new  types  of  T&E  activities. 

-  The  Center  continued  to  develop  multiple  test  tools  for 
evaluating  ASE  infrared  countermeasure  (IRCM)  systems 
and  FLFI  systems. 

-  In  addition,  the  Center  is  improving  its  electronic  warfare 
capability  with  the  development  of  the  high-power 
Portable  Range  Threat  Simulator  that  will  provide  a  more 
comprehensive  integrated  ASE  T&E  environment. 

•  The  Center  dedicated  about  two  percent  of  its  efforts  to 
providing  subject  matter  expertise  to  numerous  working 
groups  (WGs)  and  task  forces. 

The  following  activities  are  representative  of  those  conducted  by 
the  Center  during  the  past  year. 


ASE  AND  HFI  ACTIVITIES 


RESEARCH  AND  DEVELOPMENT  ACTIVITY 

Army:  Phase  2  Hostile  Fire  Tower  Test  for  Sensor  Upgrade 
Technology 

•  Sponsors:  Program  Management  Office  Aircraft  Survivability 
Equipment  (PMO-ASE)  and  Navy  Program  Executive  Officer, 
Advanced  Tactical  Aircraft  Protection  Systems  Program  Office 
(Program  Manager  Air  (PMA)-272) 

•  Activity:  The  Center  provided  Joint  Mobile  IRCM 
Testing  System  (JMITS),  ultraviolet,  and  infrared  (IR)  missile 
simulations.  The  sensors  under  test  were  the  Common  Missile 
Warning  System-Generation  3/4A,  Enhanced  Ultraviolet 
Sensor,  Passive  Infrared  Cueing  System-A,  Electro-Optic 
Missile  Sensors,  and  Advanced  Threat  Warner  (ATW). 

All  sensors  were  mounted  on  a  tower  at  a  distance  of  1.5 
kilometers  from  the  JMITS. 


•  Benefit:  The  data  collected  allowed  PMO-ASE  to  evaluate 
the  sensors  under  test  in  support  of  their  sensor  technology 
upgrade  efforts.  The  data  also  allowed  PMA-272  to  evaluate 
ATW  algorithm  updates. 

ROTARY-WING  TEST  EVENTS 

Navy:  Distributed  Aperture  Infrared  Countermeasure 

(DAIRCM)  MH-6  Test 

•  Sponsors:  Naval  Research  Laboratory,  the  U.S.  Army 
Technology  Applications  Program  Office  (TAPO),  and 
160th  Special  Operations  Aviation  Regiment  Systems 
Integration  and  Maintenance  Office 
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•  Activity:  The  Center  provided  JMITS  two-color,  IR 
simulations  and  static  IR  seekers  to  verify  the  performance  of 
the  DAIRCM  laser  as  installed  on  the  MH-6  Little  Bird. 

•  Benefit:  The  data  collected  from  this  effort  allowed  the 
sponsors  to  assess  the  performance  of  the  DAIRCM  laser 
against  threat  seekers.  The  data  also  allowed  the  Naval 
Research  Laboratory  to  identify  any  needed  improvements  for 
the  DAIRCM  system. 

Army:  Special  Operations  Aircraft  Flight  Testing  and 
Training 

•  Sponsors:  U.S.  Army  TAPO  and  160th  Special  Operations 
Aviation  Regiment  Systems  Integration  and  Maintenance 
Office 

•  Activity:  The  Center  provided  JMITS  ultraviolet  missile 
simulations  and  a  threat-representative  laser  beamrider  during 
flight  testing  of  the  MH-60M  and  MH-47G  aircraft. 

•  Benefit:  The  data  collected  from  this  effort  allowed  TAPO  to 
gather  information  on  their  advanced  ASE  suite,  the  Common 
Missile  Warning  System,  acoustic  HFI,  and  AN/AVR-2B 
laser-detecting  set. 

Navy:  CH-53E  Department  of  the  Navy  (DoN)  Large 
Aircraft  Infrared  Countermeasures  (LAIRCM)  Advanced 
Threat  Warner  (ATW)  Sensor  Upgrade  Weapons  and  Tactics 
Instructor  Flight  Test 

•  Sponsor:  Navy  Program  Executive  Officer,  Advanced 
Tactical  Aircraft  Protection  Systems  Program  Office 

•  Activity:  The  Center  provided  the  Multi-spectral  Sea  and 
Land  Target  Simulator  (MSALTS)  and  JMITS  two-color, 

IR  missile  simulators  along  with  jam  beam  radiometers, 
threat-representative  laser  beamriders,  and  a  designator  and 
rangefinder. 

•  Benefit:  The  testing  provided  a  cost-effective  test  venue  for 
collecting  critical  data  needed  to  assess  the  performance  of  the 
DoN  LAIRCM  ATW  upgrade. 

Navy:  CH-53E  DoN  LAIRCM  ATW  Sensor  Integrated  Test 
and  Evaluation  Phase  I 

•  Sponsor:  Navy  Program  Executive  Officer,  Advanced 
Tactical  Aircraft  Protection  Systems  Program  Office 

•  Activity:  The  Center  provided  threat-representative  laser 
beamriders  and  a  designator  and  rangefinder  during  flight 
testing  of  the  CH-53E  DoN  LAIRCM  ATW  system. 

•  Benefit:  The  testing  provided  a  cost-effective  test  venue  for 
collecting  critical  data  needed  to  evaluate  and  update  DoN 
LAIRCM  ATW  laser-warning  receiver  algorithms. 

Navy:  CH-53E  DoN  LAIRCM  ATW  Sensor  Integrated  Test 
and  Evaluation  Phase  II 

•  Sponsor:  Navy  Program  Executive  Officer,  Advanced 
Tactical  Aircraft  Protection  Systems  Program  Office 

•  Activity:  The  Center  provided  JMITS  and  MSALTS 
two-color,  IR  missile  simulations  and  jam  beam  radiometers 


during  flight  testing  of  the  CH-53E  DoN  LAIRCM  ATW 
system. 

•  Benefit:  The  testing  provided  a  cost-effective  test  venue  for 
collecting  critical  data  needed  to  evaluate  and  update  DoN 
LAIRCM  ATW  missile-warning  algorithms. 

Navy:  CH-53E  DoN  LAIRCM  Super  Backend  Processor 

Regression  Flight  Testing 

•  Sponsor:  Navy  Program  Executive  Officer,  Advanced 
Tactical  Aircraft  Protection  Systems  Program  Office 

•  Activity:  The  Center  provided  MSALTS  and  JMITS 
two-color,  IR  missile  simulations  and  jam  beam  radiometers 
to  support  a  proof  of  Engineering  Change  Proposal  upgrade  to 
the  DoN  LAIRCM  system. 

•  Benefit:  The  testing  provided  a  cost-effective  test  venue  for 
collecting  critical  data  needed  to  evaluate  and  update  DoN 
LAIRCM  Super  Backend  Processor  algorithms. 

Reduced  Optical  Signature  Emissions  Solution  IR 

Countermeasure  Test  VIII 

•  Sponsors:  U.S.  Army  TAPO  and  160th  Special  Operations 
Aviation  Regiment  Systems  Integration  and  Maintenance  Office 

•  Activity:  The  Center  provided  subject  matter  expertise  during 
the  IRCM  effectiveness  test  for  the  MH-60M  and  MH-47 
aircraft.  These  tests  evaluated  new  flare  CM  sequences  and 
variations  of  current  flare  CM  sequences  using  improved 
flares,  or  different  flares  within  the  sequences.  The  Center 
provided  an  independent  assessment  of  test  results  to  TAPO 
leadership. 

•  Benefit:  The  data  collected  from  this  effort  allowed  TAPO  to 
use  the  test  results  to  enhance  the  protection  of  the  MH-60M 
and  MH-47  aircraft  against  IR  Man  Portable  Air  Defense 
Systems  (MANPADS). 

NATO:  Trial  PULSATILLA  2014 

•  Sponsor:  Joint  Countermeasures  T&E  Working  Group 
(JCMT&E  WG) 

•  Activity:  The  Center  served  as  trial  manager  and  radiometric 
data  collector  during  Trial  PULSATILLA  2014  at  the 
Military  Training  Area  in  Hradiste,  Czech  Republic.  There 
were  23  organizations  representing  10  NATO  countries  that 
provided  sensors  and  ASE  systems  to  the  trial. 

•  Benefit:  Trial  PULSATILLA  2014  provided  an  opportunity  for 
NATO  Sub-Group  2  member  nations  to  expand  and  develop 
alliances,  Quick  Reaction  Assessment  (QRA)  capabilities,  and 
measure  weapon  and  ammunition  signatures,  all  of  which  will 
be  used  to  create  a  NATO  database.  Data  will  also  be  collected 
from  threat  warning  sensors  for  use  in  refining  algorithms. 

NATO:  Trial  MACE  XVI 

•  Sponsor:  JCMT&E  WG 

•  Activity:  The  Center  provided  three  analysts  to  help  process 
data  and  produce  reporting  products  during  Trial  MACE  XVI 
at  the  Military  Training  Area  in  Lest’,  Slovakia. 
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•  Benefit:  Trial  MACE  provided  an  opportunity  for  NATO 
Sub-Group  2  member  nations  to  understand  the  potential 
vulnerabilities  within  modern,  multi-function  radars  and 
integrated  air  defense  systems.  An  outcome  of  MACE  XVI 
will  be  to  develop  radio-frequency  countermeasures  advice  for 
inclusion  in  a  NATO  handbook. 

FIXED-WING  TEST  EVENTS 

Air  Force:  LAIRCM  AC-130U  Flight  Test 

•  Sponsor:  46th  Test  Wing  Test  Squadron  Defensive  Systems 
and  Mobility  Directorate,  Air  Force  Life  Cycle  Management 
Center 

•  Activity:  The  Center  provided  JMITS  missile  simulators 
and  personnel  to  perform  two-color,  IR  simulations  to  collect 
system  response  data  for  assessing  the  LAIRCM  system  as 
installed  on  the  AC-130U.  The  test  was  conducted  at  Eglin  Air 
Force  Base  (AFB),  Florida. 

•  Benefit:  The  testing  provided  the  Air  Force  with  a 
cost-effective  test  venue  for  collecting  critical  data  needed  to 
assess  performance  of  the  LAIRCM  system  as  installed  on  a 
new  platform,  the  AC-130U. 

Air  Force:  LAIRCM  MC-130H  Flight  Test 

•  Sponsors:  46th  Test  Wing  Test  Squadron  Defensive  Systems 
and  Mobility  Directorate,  Air  Force  Life  Cycle  Management 
Center 

•  Activity:  The  Center  provided  JMITS  missile  simulators 
and  personnel  to  perform  two-color,  IR  simulations  to  collect 
system  response  data  for  assessing  the  LAIRCM  system 

as  installed  on  the  MC-130H.  The  tests  were  conducted  at 
Eglin  AFB,  Florida. 

•  Benefit:  The  testing  provided  the  Air  Force  with  critical 
data  needed  to  assess  performance  of  the  LAIRCM  system  as 
installed  on  the  MC-130H. 

Air  Force:  LAIRCM  System  Processor  Replacement  Flight 

Test 

•  Sponsors:  46th  Test  Wing  Test  Squadron  Defensive  Systems 
and  Mobility  Directorate,  Air  Force  Life  Cycle  Management 
Center 


•  Activity:  The  Center  provided  JMITS  missile  simulators 
and  personnel  to  perform  two-color,  IR  simulations  to  collect 
system  response  data  for  assessing  the  upgraded  system 
software  with  the  new  system  processor.  The  tests  were 
conducted  at  Eglin  AFB,  Florida. 

•  Benefit:  The  testing  provided  the  Air  Force  with  critical 
data  needed  to  assess  performance  of  the  upgraded  LAIRCM 
system. 

Air  Force:  QF-16  Live  Fire 

•  Sponsors:  Air  Force  Operational  Test  and  Evaluation 
Command,  Detachment  2 

•  Activity:  The  Center  provided  two  remote-launch  systems 
and  operators  to  launch  surface-to-air  missiles  at  a  QF-16  to 
demonstrate  the  installed  Vector  Scoring  System  capabilities. 

•  Benefit:  The  results  of  this  live  fire  test  will  help  support  a 
Full-Rate  Production  decision. 

Air  Force:  F-35  IR  Countermeasure  Test 

•  Sponsors:  Joint  Strike  Fighter  Program  Office 

•  Activity:  The  Center  provided  personnel  to  support  the 
Missile  and  Space  Intelligence  Center  with  the  operation  of 
reactive-  and  preemptive-configured  IR  seekers. 

•  Benefit:  The  data  collected  from  this  effort  allowed  the 
sponsors  to  assess  the  performance  of  various  IRCM  flare 
sequences. 

Navy:  C-40  Guardian  Pod  Flight  Testing 

•  Sponsor:  Navy  Program  Executive  Officer,  Advanced 
Tactical  Aircraft  Protection  Systems  Program  Office 

•  Activity:  The  Center  provided  JMITS  two-color,  IR  missile 
simulations  during  flight  testing  of  the  C-40  ASE.  The 
Center  provided  all  data  collected  to  the  sponsors  for  their 
assessments. 

•  Benefit:  The  testing  provided  the  critical  data  needed  to 
support  a  fleet  introduction  decision  for  the  Guardian  Pod  as 
installed  on  the  U.S.  Navy  C-40A  aircraft. 


PGW  CM  ACTIVITIES 


Army:  Dazzler 

•  Sponsor:  U.S.  Army  Research,  Development  and  Engineering 
Command,  Armament  Research,  Development  and 
Engineering  Center 

•  Activity:  The  Center  coordinated,  directed,  conducted,  and 
collected  data  for  this  event.  The  test  was  performed  to 
characterize  and  evaluate  the  effectiveness  of  new  pyrotechnic 
CM. 

•  Benefit:  The  data  collected  will  be  used  to  evaluate  the 
effectiveness  of  the  CM  as  intended  and  to  adjust  the  CM 
formulation  if  required. 


Air  Force:  Small  Diameter  Bomb  (SDB)  II  Obscurants  Test 

•  Sponsor:  SDB  II  Program  Office 

•  Activity:  The  Center,  in  conjunction  with  the  sponsors,  the 
46th  Test  Wing  and  782nd  Test  Squadron,  Eglin  AFB,  Florida, 
coordinated,  directed,  and  conducted  the  captive-carry  test  of 
this  air-to-ground  missile  system.  The  Center  provided  some 
obscurants  and  deployed  all  obscurants. 

•  Benefit:  The  field  test  provided  an  opportunity  for  the 
SDB  II  Program  Office  to  determine  seeker  performance 
against  static-  and  moving-ground  mobile  targets  in  obscurant 
environments. 
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National  Ground  Intelligence  Center:  Grackle  Oyster 

•  Sponsor:  National  Ground  Intelligence  Center  and  SDB 
Program  Office 

•  Activity:  The  Center,  in  conjunction  with  the  sponsors,  the 
46th  Test  Wing,  and  782nd  Test  Squadron,  Eglin  AFB,  Florida, 


coordinated,  directed,  and  conducted  the  field  exploitation  of 
this  foreign  battlefield  obscurant. 

•  Benefit:  This  event  provided  field  characterization  data  for 
use  in  the  modeling  and  simulation  of  the  performance  of 
weapon  system  sensors. 


CM-BASED  PRE-DEPLOYMENTTRAINING  FOR  SERVICE  MEMBER  EXERCISES 


Surface  Attack  Training  -  Nellis  AFB,  Nevada 
160th  Special  Operations  Aviation  Regiment  Radio 
Frequency  Training  -  Hill  AFB,  Utah 
Joint  Forcible  Entry  -  Nellis  AFB,  Nevada 
Advanced  Integration  Exercise  -  Nellis  AFB,  Nevada 
Joint  Readiness  Training  Center  Training 
Support  -  Fort  Polk,  Louisiana 
Emerald  Warrior  -  Hurlburt  Field,  Florida 
10th  Aviation  Brigade,  6th  Squadron,  6  Cavalry 
Training  -  Fort  Polk,  Louisiana 
509th  Weapons  Squadron  KC-135 
Support  -  Roswell,  New  Mexico 

•  Sponsors:  Various 

•  Purpose:  The  Center’s  equipment  and  personnel  provided 
a  simulated  threat/CM  environment  and  subject  matter 


expertise  to  observe  aircraft  sensor/ASE  systems  and  crew 
reactions  to  this  environment.  Specifically,  the  Center 
emphasized  simulated  MANPADS  and  radio-frequency 
threat  engagements  for  participating  aircraft.  Additionally, 
the  Center  provided  MANPADS  capabilities  and  limitations 
briefings  to  pilots  and  crews  and  conducted  “hands-on” 
training  at  the  end  of  the  briefings. 

•  Benefit:  Provided  realism  to  the  training  threat  environment 
for  the  Service  member  pilots  and  crews  to  facilitate 
understanding  and  use  of  CM  equipment,  especially  ASE. 
The  Center  provided  collected  data  to  the  trainers  for 
assisting  units  in  the  development/refinement  of  techniques, 
tactics,  and  procedures  to  enhance  survivability. 


SURVIVABILITY  INITIATIVES 


Hostile  Fire  Signature  (HSIG)  Model 

The  Center  led  development  of  the  HSIG  model  to  support 
HFI  T&E  and  modeling  efforts.  The  HSIG  Model  project, 
sponsored  by  the  Threat  Systems  WG  with  oversight  by  the 
Test  and  Evaluation  Threat  Resource  Activity,  has  developed 
a  physics-based  electro-optic  model  that  produces  signatures 
for  the  12.7  mm  Armor  Piercing  Incendiary  Tracer  round  and  a 
rocket-propelled  grenade  (RPG  7)  tracer  and  hardbody.  Final 
model  validation  report  was  completed  in  FY14.  Additional 
support  is  being  sought  to  incorporate  RPG  back  blast  and 
small-arm  muzzle  flash  features  to  the  models. 

Enhanced  Missile  Signature  (EMSIG)  Model 
The  Center  was  instrumental  in  identifying  the  need  to  fund  the 
development  and  integration  of  six  additional  missile  signature 
models  along  with  the  Army’s  Re -programming  Analysis 
Team  and  Services’  Program  Offices  at  the  Common  IRCM 
missile  summit.  These  additional  models  will  provide  all  of 
the  Services’ ASE  and  countermeasure  programs  with  a  more 
comprehensive  threat  signature  database. 

Joint  Countermeasures  T&E  Working  Group 
(JCMT&EWG) 

The  JCMT&E  WG  is  co-chartered  by  DOT&E  and 
DASD(DT&E)  to  improve  the  integration  of: 

•  Aircraft  self-protection  and  countermeasure  developments 

•  Live-fire  threat  weapon  open-air  T&E 

•  Developmental  and  OT&E 

•  Development  of  standardized  test  methodologies 


•  Common  instrumentation  and  standards 

•  Overseas  threat  and  air  electronic  warfare  systems 
performance  and  effectiveness  data 

The  JCMT&E  WG  includes  DOT&E,  DASD(DT&E),  all 
four  of  the  U.S.  Services,  Australia,  Canada,  Great  Britain, 

New  Zealand,  and  NATO  Air  Force  Armaments  Group 
Sub-Group  2  as  members  of  a  coalition  warfare  sub-WG.  The 
group  is  tasked  with  actively  seeking  mutually-beneficial 
T&E  opportunities  to  measure  performance  and  suitability 
data  necessary  to  provide  relevant  operational  information 
to  deploying  joint/coalition  Service  members  and  for  U.S. 
acquisition  decision  makers.  Specific  efforts  included  the 
following. 

The  JCMT&E  WG,  in  the  capacity  of  the  Chainnan  of  the 
eight-year  bilateral  ASE  Cooperative  Test  and  Evaluation 
Project  Arrangement  Steering  Committee,  worked  with  Great 
Britain  to  ensure  smooth  and  highly-effective  testing  of 
ASE.  The  United  States  and  Great  Britain  have  developed 
and  successfully  implemented  three  WGs  in  order  to  more 
effectively  manage  the  growing  level  of  efforts.  The 
two  nations’  defense  organizations,  ASE  Program  Offices, 
developmental  testing,  operational  testing,  and  LFT&E  agencies 
have  been  able  to  collaborate  on  common  test  equipment 
methodologies  and  measure  threat  missiles,  guns,  and  rockets’ 
effects  on  ASE  using  actual  threat  gunners,  weapons,  and 
environmental  data  that  will  continue  to  improve  Service 
member  survivability. 
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The  JCMT&E  WG,  in  the  capacity  of  the  Chairman  of  the 
10-year  bilateral  ASE  Cooperative  Test  and  Evaluation 
Project  Arrangement  Steering  Committee,  worked  with 
Australia  to  ensure  smooth  and  highly-effective  testing  on 
both  sides  of  the  Pacific.  The  United  States  and  Australia 
developed  and  successfully  implemented  three  WGs  to  more 
effectively  manage  the  growing  level  of  efforts.  As  a  result, 
the  Center  participated  in  the  planning  of  the  Australian 
Trial  OXIDIZER  II  and  other  data  collection  opportunities  using 
the  Marine  Rotational  Force  in  Darwin,  Northern  Territory.  These 
data  will  be  used  to  improve  U.S.  threat  detection  algorithms 
while  reducing  both  nations’  test  costs. 

The  JCMT&E  WG  was  the  U.S.  Technical  Advisor  to 
the  official  negotiations  of  the  Multinational  Test  and 
Evaluation  Program  memorandum  of  understanding  with 
Australia,  Canada,  Great  Britain,  New  Zealand,  and  the 
United  States.  In  support  of  high-level  NATO  multinational 
approaches  initiatives  and  DOT&E  initiatives  to  NATO,  the 
Center  developed,  organized,  and  conducted  a  highly-successful, 
10-nation  NATO  QRA  in  the  Czech  Republic.  The  calibrated 
data  and  expert  analysis  measured  has  been  hailed  as  the  model 
for  NATO  to  expand  use  for  future  QRAs.  Due  to  the  Center’s 
efforts,  the  NATO  National  Armaments  Directors  Representative 
designated  the  Defensive  Aids  Suite  effort  a  Smart  Defence  Tier  1 
project. 


The  JCMT&E  WG  worked  with  the  Office  of  the  Deputy 
Assistant  Secretary  of  the  Army  for  Defense  Exports  and 
Cooperation;  the  Navy  International  Program  Office;  the  Deputy 
Under  Secretary  of  the  Air  Force,  International  Affairs;  and  the 
Services’  program  managers  to  spear-head  the  development  of 
the  four-nation  Aircraft  Electronic  Warfare  Cooperative  Test 
and  Evaluation  Project  Arrangement  that  will  be  negotiated 
in  November  2014.  This  effort  will  coalesce  many  of  the 
redundant  testing  conducted  by  Australia,  Canada,  Great 
Britain,  and  the  United  States  while  significantly  expanding 
performance  and  suitability  data  at  reduced  cost  for  all  four 
nations. 

The  JCMT&E  WG  worked  with  NATO  Air  Force  Armaments 
Group  Sub-Group  2  to  develop,  plan,  and  conduct  a  major 
threat  data  collection  effort  in  the  Czech  Republic.  Joined 
by  10  nations  and  over  80  technicians,  the  Center  managed 
the  firing  of  operationally  significant  types  and  measured  the 
performance  of  surface-to-air  missiles,  anti-tank  weapons, 
rockets,  and  Soldier-fired  weapons  in  Trial  PULSATILLA. 

This  trial  provided  the  United  States  and  our  allies  with 
substantive  data  for  use  by  threat-warning  systems’  developers 
and  improvements  in  tactics,  techniques,  and  procedures  by  our 
Service  members. 


THREAT  SIMULATOR  TEST  AND  EVALUATION  TOOLS 


The  Center  has  continued  to  develop  tools  for  T&E  of  IRCM 
systems  funded  by  the  USD(AT&L),  Test  Resource  Management 
Center,  and  Central  Test  and  Evaluation  Investment  Program. 
Currently,  the  Center  is  leading  the  development  of  MSALTS  and 
the  Joint  Standard  Instrumentation  Suite  (JSIS). 

•  The  MSALTS  is  a  small,  mobile  missile  simulator  that  can  fire 
while  moving  and  simulate  all  current  tier-one  missile  threats. 
The  Center  has  designed  the  MSALTS  to  provide  simulated 
signatures  for  the  new  and  more  capable  missile-warning 
systems,  such  as  LA1RCM  Next  Generation,  DoN  LAIRCM, 
and  the  DoN  ATW.  The  Center  initiated  development  of  the 
first  two  systems  in  FY11  and  the  third  system  in  FY12.  The 
developer  completed  fabrication,  assembly,  and  integration, 
and  executed  government  acceptance  testing  of  the  first 
MSALTS  system  in  April  2014.  Acceptance  testing  of 
subsequent  units  completed  in  October  2014. 

•  The  JSIS  is  a  transportable,  fully-integrated  instrumentation 
suite  that  will  be  utilized  for  collecting  signature, 
Time-Space-Position  Information,  acoustic,  and  related 
metadata  of  threat  missile  and  hostile  fire  munitions.  The 
transportability  of  JSIS  will  allow  it  to  be  used  both  in  the 
United  States  and  abroad  with  the  intent  of  reducing  costs 
and  expanding  the  types  of  threats  available  in  the  United 
States.  JSIS  data  collected  during  these  live  fire  events  will 
be  used  to  support  ASE  systems  development,  modeling  and 
simulation  activities,  T&E  ground  truth  data,  and  anomaly 
investigation.  All  data  collected  from  JSIS  will  be  calibrated, 


measured,  and  stored  according  to  the  standards  defined  by 
the  Joint  Tactical  Missile  Signatures  Handbook  and  will  be 
available  to  the  ASE  community.  The  JSIS  has  been  endorsed 
by  the  U.S.  Navy  (PMA  -  272),  Army  (PMO  -  ASE),  and  the 
Air  Force  (LAIRCM  System  Program  Office)  and  will  be  an 
integral  part  in  each  Program  Office’s  ASE  development.  In 
July  2013,  the  JSIS  was  selected  as  a  “Resource  Enhancement 
Project  New  Start”  project  and  received  FY14  funding  from 
the  Test  Resource  Management  Center  and  Central  Test 
and  Evaluation  Investment  Program.  In  FY14,  the  Center, 
partnered  with  the  Arnold  Engineering  Development  Center, 
actively  created  program  plans,  refined  requirements  from 
the  ASE  T&E  community,  created  and  refined  a  concept  of 
operation,  and  began  identifying  specific  instrumentation  that 
meets  JSIS  requirements.  A  successful  Critical  Design  Review 
was  conducted  in  May  2014. 

Additionally,  as  a  result  of  an  internal  electronic  warfare  study 
conducted  by  the  Center  in  FY13,  and  the  increasing  demands 
for  test  tools  that  support  multi-spectral,  integrated  ASE  threat 
environments,  the  Center  internally  funded  the  procurement  of 
two  radio-frequency  threat  emitters.  A  low-powered  Portable 
Range  Threat  Simulator  system  will  be  delivered  in  early  FY15 
and  a  high-powered  Portable  Range  Threat  Simulator  capability 
is  scheduled  to  be  delivered  in  FY16.  These  systems  are  being 
designed  to  replicate  short-range  acquisition  and  targeting  radar 
systems. 


CCM  361 


F  Y 1 4  CENTER  FOR  COUNTERMEASURES 


THREAT  LIVE  WEAPON  FIRE  TEST  AND  EVALUATION  TOOLS 


The  Center  has  continued  to  develop  tools  for  the  T&E 
community  for  threat,  live-fire  IRCM  testing.  In  FY14  a  new 
remote-missile  launcher  was  developed  by  Missile  and  Space 
Intelligence  Center  for  the  Center.  This  launcher  system  was 


developed  to  support  remote  firing  of  larger  vehicle-launched  IR 
surface-to-air  missiles.  The  system  was  delivered  and  is  expected 
to  be  operationally  verified  during  live  fire  acceptance  testing  in 
early  FY15. 
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